diff --git a/docs/packages/compono-testdoubles.md b/docs/packages/compono-testdoubles.md index 3a7e6e4..cf5e0ce 100644 --- a/docs/packages/compono-testdoubles.md +++ b/docs/packages/compono-testdoubles.md @@ -17,10 +17,15 @@ dotnet add package Compono.TestDoubles --prerelease `Compono.TestDoubles` is not a general-purpose mocking framework — see [ADR-0042](../adr/0042-compono-owned-source-generated-test-doubles.md)'s -Non-Goals. If you need call verification, argument matchers, or a familiar -runtime-proxy substitute, use -[`Compono.NSubstitute`](compono-nsubstitute.md) instead; the two packages -are not mutually exclusive. +Non-Goals — but current generated doubles do support `Configure()`, +`Verify()`, literal equality matching, `Match.Any()`, +`Match.Is(predicate)`, argument-filtered `Never()`/`Once()`/`Exactly(n)`, +and multi-entry argument-distinguished response configuration for eligible +member shapes. Use [`Compono.NSubstitute`](compono-nsubstitute.md) when you +intentionally want a familiar runtime-proxy substitute or a capability still +outside generated-double support, such as invocation-aware callbacks, true +argument capture, call-order verification, or partial/strict substitutes; +the two packages are not mutually exclusive. ## Compile-time opt-in @@ -71,10 +76,12 @@ service.Repository.Configure().CountAsync().Returns(Task.FromResult(4)); `using` or a generation failure — see [ADR-0043 Amendment 3 Finding C](../adr/0043-compono-generated-test-doubles-design.md#amendment-3-2026-08-13-public-cross-assembly-state-contract-overloadname-collision-diagnostics-documented-multi-assembly-registry-limitation). - **Per-member `.Returns(...)`/`.Throws(...)`** — configure a method or - property's behavior; last configuration wins (calling `.Returns(...)` - after an earlier `.Throws(...)` on the same member clears the exception, - and vice versa). Configuration is member-level and **argument- - independent** — there are no argument matchers in v1. + property's behavior. A zero-argument `Configure().Member()` applies to + every call to that member. For eligible parameterized members, + `Configure().Member(...)` accepts literal equality arguments, + `Match.Any()`, and `Match.Is(predicate)`; multiple argument- + distinguished configurations can coexist, with the most recently + registered matching entry winning. - **Deterministic defaults for unconfigured members** — primitives, nullable references, `Task`/`Task`, `ValueTask`/`ValueTask`, and known collection shapes (arrays, `List`, `Dictionary`, @@ -664,4 +671,6 @@ An unsupported member shape is a compile-time diagnostic - [Providers](../concepts/providers.md) — where the generated-double provider sits in the resolution pipeline. - [`Compono.NSubstitute`](compono-nsubstitute.md) — the runtime-proxy - alternative, for call verification/argument matchers. + alternative, for capabilities still outside generated-double support + (for example invocation-aware callbacks, true argument capture, + call-order verification, or partial/strict substitutes). diff --git a/docs/reference/api/Compono/Compono.CallVerifier.CallVerifier(int,string).md b/docs/reference/api/Compono/Compono.CallVerifier.CallVerifier(int,string).md index c3ca9c3..34703b5 100644 --- a/docs/reference/api/Compono/Compono.CallVerifier.CallVerifier(int,string).md +++ b/docs/reference/api/Compono/Compono.CallVerifier.CallVerifier(int,string).md @@ -3,10 +3,9 @@ ## CallVerifier\(int, string\) Constructor -Asserts how many times a generated test double's member was called, backed by -[ConfiguredCallCount](Compono.ReturnConfig_T_.ConfiguredCallCount.md 'Compono\.ReturnConfig\\.ConfiguredCallCount')\. Deliberately minimal \- [Never\(\)](Compono.CallVerifier.Never().md 'Compono\.CallVerifier\.Never\(\)')/ -[Once\(\)](Compono.CallVerifier.Once().md 'Compono\.CallVerifier\.Once\(\)')/[Exactly\(int\)](Compono.CallVerifier.Exactly(int).md 'Compono\.CallVerifier\.Exactly\(int\)') only, no argument matchers, no call\-order verification, -per ADR\-0044 Requirement 3\. +Performs terminal call\-count assertions after any generated member and argument filtering has +already been applied\. Deliberately minimal \- [Never\(\)](Compono.CallVerifier.Never().md 'Compono\.CallVerifier\.Never\(\)')/[Once\(\)](Compono.CallVerifier.Once().md 'Compono\.CallVerifier\.Once\(\)')/ +[Exactly\(int\)](Compono.CallVerifier.Exactly(int).md 'Compono\.CallVerifier\.Exactly\(int\)') only, with no call\-order verification, per ADR\-0044 Requirement 3\. ```csharp public CallVerifier(int observedCount, string memberDescription); @@ -17,7 +16,7 @@ public CallVerifier(int observedCount, string memberDescription); `observedCount` [System\.Int32](https://learn.microsoft.com/en-us/dotnet/api/system.int32 'System\.Int32') -How many times the member's dispatch body actually ran\. +How many matching calls the generated verification surface observed\. diff --git a/docs/reference/api/Compono/Compono.CallVerifier.md b/docs/reference/api/Compono/Compono.CallVerifier.md index c614274..1dee3c0 100644 --- a/docs/reference/api/Compono/Compono.CallVerifier.md +++ b/docs/reference/api/Compono/Compono.CallVerifier.md @@ -3,10 +3,9 @@ ## CallVerifier Struct -Asserts how many times a generated test double's member was called, backed by -[ConfiguredCallCount](Compono.ReturnConfig_T_.ConfiguredCallCount.md 'Compono\.ReturnConfig\\.ConfiguredCallCount')\. Deliberately minimal \- [Never\(\)](Compono.CallVerifier.Never().md 'Compono\.CallVerifier\.Never\(\)')/ -[Once\(\)](Compono.CallVerifier.Once().md 'Compono\.CallVerifier\.Once\(\)')/[Exactly\(int\)](Compono.CallVerifier.Exactly(int).md 'Compono\.CallVerifier\.Exactly\(int\)') only, no argument matchers, no call\-order verification, -per ADR\-0044 Requirement 3\. +Performs terminal call\-count assertions after any generated member and argument filtering has +already been applied\. Deliberately minimal \- [Never\(\)](Compono.CallVerifier.Never().md 'Compono\.CallVerifier\.Never\(\)')/[Once\(\)](Compono.CallVerifier.Once().md 'Compono\.CallVerifier\.Once\(\)')/ +[Exactly\(int\)](Compono.CallVerifier.Exactly(int).md 'Compono\.CallVerifier\.Exactly\(int\)') only, with no call\-order verification, per ADR\-0044 Requirement 3\. ```csharp public readonly struct CallVerifier @@ -14,7 +13,7 @@ public readonly struct CallVerifier | Constructors | | | :--- | :--- | -| [CallVerifier\(int, string\)](Compono.CallVerifier.CallVerifier(int,string).md 'Compono\.CallVerifier\.CallVerifier\(int, string\)') | Asserts how many times a generated test double's member was called, backed by [ConfiguredCallCount](Compono.ReturnConfig_T_.ConfiguredCallCount.md 'Compono\.ReturnConfig\\.ConfiguredCallCount')\. Deliberately minimal \- [Never\(\)](Compono.CallVerifier.Never().md 'Compono\.CallVerifier\.Never\(\)')/ [Once\(\)](Compono.CallVerifier.Once().md 'Compono\.CallVerifier\.Once\(\)')/[Exactly\(int\)](Compono.CallVerifier.Exactly(int).md 'Compono\.CallVerifier\.Exactly\(int\)') only, no argument matchers, no call\-order verification, per ADR\-0044 Requirement 3\. | +| [CallVerifier\(int, string\)](Compono.CallVerifier.CallVerifier(int,string).md 'Compono\.CallVerifier\.CallVerifier\(int, string\)') | Performs terminal call\-count assertions after any generated member and argument filtering has already been applied\. Deliberately minimal \- [Never\(\)](Compono.CallVerifier.Never().md 'Compono\.CallVerifier\.Never\(\)')/[Once\(\)](Compono.CallVerifier.Once().md 'Compono\.CallVerifier\.Once\(\)')/ [Exactly\(int\)](Compono.CallVerifier.Exactly(int).md 'Compono\.CallVerifier\.Exactly\(int\)') only, with no call\-order verification, per ADR\-0044 Requirement 3\. | | Methods | | | :--- | :--- | diff --git a/docs/reference/api/Compono/Compono.md b/docs/reference/api/Compono/Compono.md index 2c3c6d7..96ab340 100644 --- a/docs/reference/api/Compono/Compono.md +++ b/docs/reference/api/Compono/Compono.md @@ -32,7 +32,7 @@ | Structs | | | :--- | :--- | -| [CallVerifier](Compono.CallVerifier.md 'Compono\.CallVerifier') | Asserts how many times a generated test double's member was called, backed by [ConfiguredCallCount](Compono.ReturnConfig_T_.ConfiguredCallCount.md 'Compono\.ReturnConfig\\.ConfiguredCallCount')\. Deliberately minimal \- [Never\(\)](Compono.CallVerifier.Never().md 'Compono\.CallVerifier\.Never\(\)')/ [Once\(\)](Compono.CallVerifier.Once().md 'Compono\.CallVerifier\.Once\(\)')/[Exactly\(int\)](Compono.CallVerifier.Exactly(int).md 'Compono\.CallVerifier\.Exactly\(int\)') only, no argument matchers, no call\-order verification, per ADR\-0044 Requirement 3\. | +| [CallVerifier](Compono.CallVerifier.md 'Compono\.CallVerifier') | Performs terminal call\-count assertions after any generated member and argument filtering has already been applied\. Deliberately minimal \- [Never\(\)](Compono.CallVerifier.Never().md 'Compono\.CallVerifier\.Never\(\)')/[Once\(\)](Compono.CallVerifier.Once().md 'Compono\.CallVerifier\.Once\(\)')/ [Exactly\(int\)](Compono.CallVerifier.Exactly(int).md 'Compono\.CallVerifier\.Exactly\(int\)') only, with no call\-order verification, per ADR\-0044 Requirement 3\. | | [CompositionProviderRequest](Compono.CompositionProviderRequest.md 'Compono\.CompositionProviderRequest') | A composition request, as seen by a public [ICompositionValueProvider](Compono.ICompositionValueProvider.md 'Compono\.ICompositionValueProvider') \- decoupled from the engine's own internal `Compono.CompositionRequest` \(no path, no shared\-scope flag, no pipeline plumbing a provider author has no legitimate use for\)\. See `docs/adr/0024-public-provider-extensibility-model.md`\. | | [CompositionProviderResult](Compono.CompositionProviderResult.md 'Compono\.CompositionProviderResult') | What an [ICompositionValueProvider](Compono.ICompositionValueProvider.md 'Compono\.ICompositionValueProvider') reports for one [CompositionProviderRequest](Compono.CompositionProviderRequest.md 'Compono\.CompositionProviderRequest')\. | | [CompositionRequestDescriptor](Compono.CompositionRequestDescriptor.md 'Compono\.CompositionRequestDescriptor') | The compact, compile\-time\-constructible value a generated [ICompositionPlan<T>](Compono.ICompositionPlan_T_.md 'Compono\.ICompositionPlan\') passes to [Resolve<TValue>\(CompositionRequestDescriptor\)](Compono.ICompositionContext.Resolve.md#Compono.ICompositionContext.Resolve_TValue_(Compono.CompositionRequestDescriptor) 'Compono\.ICompositionContext\.Resolve\\(Compono\.CompositionRequestDescriptor\)') for one constructor parameter or required member \- or, for a test\-framework integration composing a [CompositionRow](Compono.CompositionRow.md 'Compono\.CompositionRow'), one of a test method's own parameters \([TestParameter](Compono.CompositionRequestKind.md#Compono.CompositionRequestKind.TestParameter 'Compono\.CompositionRequestKind\.TestParameter')\)\. | diff --git a/docs/research/0012-aws-secrets-manager-provider-compono-skill-regression.md b/docs/research/0012-aws-secrets-manager-provider-compono-skill-regression.md new file mode 100644 index 0000000..47aa9b3 --- /dev/null +++ b/docs/research/0012-aws-secrets-manager-provider-compono-skill-regression.md @@ -0,0 +1,113 @@ +# [RESEARCH-0012] AWS Secrets Manager Provider Compono Skill Regression + +**Status:** Done + +**Date:** 2026-08-27 + +## Scope + +This records two focused follow-up findings from the real AWS Secrets Manager +Provider migration from AutoFixture/AutoFixture.Xunit3/NSubstitute to +`Compono.XunitV3` + `Compono.TestDoubles`. + +This is not a product design record and creates no ADR. It applies the existing +ADR-0029/ADR-0042 evidence rules to classify what the migration actually showed. + +## Finding A: installed skill missed shipped argument matching support + +During the migration, the installed Compono skill initially led the agent to +write hand-made recording fakes for `IAmazonSecretsManager`, +`IConfigurationBuilder`, and `ILoggerFactory`, asserting that +`Compono.TestDoubles` intentionally did not support argument matchers/capture. +That was incorrect for ordinary matching and filtered verification. + +The final migration used the shipped `Compono.TestDoubles` surface directly: + +```csharp +configurationBuilder.Configure() + .Add(Match.Any()) + .Returns(configurationBuilder); + +configurationBuilder.Verify() + .Add(Match.Is(predicate)) + .Once(); +``` + +and: + +```csharp +secretsManager.Configure() + .GetSecretValueAsync( + Match.Is(predicate), + Match.Any()) + .Returns(Task.FromResult(response)); +``` + +The migrated suite passed across `net8.0`, `net9.0`, `net10.0`, and `net11.0` +(62/62 on each TFM), with AutoFixture/NSubstitute/`Compono.NSubstitute` absent +from direct and transitive dependencies. + +Classification: **skill/docs regression**, not a runtime `Compono.TestDoubles` +capability gap. Current `Compono.TestDoubles` supports literal equality +matching, `Match.Any()`, `Match.Is(predicate)`, argument-filtered +`Never()`/`Once()`/`Exactly(n)`, and multi-entry argument-distinguished response +configuration for eligible member shapes. True capture/callback behavior remains +a separate boundary. + +## Finding B: `TestConfigurationProvider` is existing project-local test architecture + +The migration retained a local `TestConfigurationProvider : ConfigurationProvider`. +This is not new evidence for abstract-class generated doubles. + +Pre-migration evidence from the AWS Secrets Manager Provider test project: + +```csharp +public class ConfigurationProviderSpecimenBuilder : ISpecimenBuilder +{ + public object Create(object request, ISpecimenContext context) + { + if (request is Type type && type == typeof(ConfigurationProvider)) + { + return new TestConfigurationProvider(); + } + + return new NoSpecimen(); + } +} +``` + +```csharp +public class TestConfigurationProvider : ConfigurationProvider +{ + public override void Set(string key, string value) + { + Data[key] = value; + } +} +``` + +The fixture registered that specimen builder before `AutoNSubstituteCustomization`: + +```csharp +fixture.Customizations.Add(new ConfigurationProviderSpecimenBuilder()); + +fixture.Customize(new AutoNSubstituteCustomization +{ + GenerateDelegates = true +}); +``` + +Search of the pre-migration state found no `Substitute.For()` +and no NSubstitute-backed substitution of `ConfigurationProvider`. + +Classification: **existing project-local test architecture**, not a Compono +capability gap. Compono preserves the same design through: + +```csharp +builder.Register(_ => new TestConfigurationProvider()); +``` + +ADR-0042 Amendment 2 does not apply because there is no evidenced +`Compono.NSubstitute`/NSubstitute-can vs. `Compono.TestDoubles`-cannot +replacement case. No roadmap item, ADR, or abstract-class support design is +created from this evidence. diff --git a/skills/compono-evals/benchmarks/2026-08-27/benchmark.json b/skills/compono-evals/benchmarks/2026-08-27/benchmark.json new file mode 100644 index 0000000..5bbce3a --- /dev/null +++ b/skills/compono-evals/benchmarks/2026-08-27/benchmark.json @@ -0,0 +1,74 @@ +{ + "skill_name": "compono", + "date": "2026-08-27", + "runner": "pi --print --no-context-files --no-skills --skill --thinking minimal", + "scope": "Focused AWS Secrets Manager Provider TestDoubles matching/capture regression evals only", + "baseline": { + "type": "old_skill", + "snapshot_commit": "644a5ad", + "note": "Full skills/compono directory (SKILL.md + references/) snapshotted at 644a5ad, the commit before this PR's fix (5139507, caae88c). All 8 runs (4 evals x with_skill/old_skill) launched together in one batch so with_skill and old_skill are genuinely paired, per skill-creator's workflow." + }, + "results": [ + { + "eval_id": 28, + "name": "aws-iconfigurationbuilder-argument-filtered-verification", + "configuration": "with_skill", + "passed": true, + "evidence": "Output uses Configure().Add(Match.Any()).Returns(configurationBuilder) and Verify().Add(Match.Is(...)).Once(), maps Arg.Is to Match.Is, and does not recommend a recording fake." + }, + { + "eval_id": 28, + "name": "aws-iconfigurationbuilder-argument-filtered-verification", + "configuration": "old_skill", + "passed": false, + "evidence": "Output claims Compono.TestDoubles cannot express the argument predicate, uses zero-arg Configure().Add()/Verify().Add().Once(), and says asserting the argument type still requires keeping the test on NSubstitute." + }, + { + "eval_id": 29, + "name": "aws-iamazonsecretsmanager-argument-matched-configuration", + "configuration": "with_skill", + "passed": true, + "evidence": "Output uses Configure().GetSecretValueAsync(Match.Is(...), Match.Any()).Returns(...) and maps Arg.Is/Arg.Any directly to Match.Is/Match.Any." + }, + { + "eval_id": 29, + "name": "aws-iamazonsecretsmanager-argument-matched-configuration", + "configuration": "old_skill", + "passed": false, + "evidence": "Output drops the argument matcher entirely, uses parameterless Configure().GetSecretValueAsync().Returns(...), and states 'Compono.TestDoubles is argument-independent' explicitly." + }, + { + "eval_id": 30, + "name": "adversarial-nsubstitute-vocabulary-trap", + "configuration": "with_skill", + "passed": true, + "evidence": "Output maps Arg.Is/Arg.Any/Received/DidNotReceive to Match.Is/Match.Any/Verify().Once/Exactly/Never and rejects recording fakes solely for NSubstitute vocabulary." + }, + { + "eval_id": 30, + "name": "adversarial-nsubstitute-vocabulary-trap", + "configuration": "old_skill", + "passed": false, + "evidence": "Output says 'Arg.Any/Arg.Is usually disappears because generated doubles are argument-independent' and lists argument-specific returns/verification as behavior Compono.TestDoubles does not provide, contradicting the expected Arg.Is->Match.Is / Arg.Any->Match.Any mapping." + }, + { + "eval_id": 31, + "name": "true-callback-capture-boundary", + "configuration": "with_skill", + "passed": true, + "evidence": "Output says ordinary matching/filtering is supported but invocation-aware callbacks/capture/delegate invocation are outside Match; recommends local fake or NSubstitute seam." + }, + { + "eval_id": 31, + "name": "true-callback-capture-boundary", + "configuration": "old_skill", + "passed": false, + "evidence": "Output correctly recommends a local fake for the callback boundary, but does so by claiming 'no argument matchers' / 'no argument-aware behavior' exist at all in Compono.TestDoubles, contradicting the expectation that ordinary matching/argument-filtered verification is supported." + } + ], + "summary": { + "with_skill": { "passed": 4, "total": 4, "pass_rate": 1.0 }, + "old_skill": { "passed": 0, "total": 4, "pass_rate": 0.0 }, + "old_eval_regression_status": "not_rerun_full_benchmark_claude_weekly_limit; existing eval definitions 1-27 unchanged" + } +} diff --git a/skills/compono-evals/benchmarks/2026-08-27/benchmark.md b/skills/compono-evals/benchmarks/2026-08-27/benchmark.md new file mode 100644 index 0000000..4608ee1 --- /dev/null +++ b/skills/compono-evals/benchmarks/2026-08-27/benchmark.md @@ -0,0 +1,61 @@ +# Compono skill regression benchmark — 2026-08-27 + +Focused regression run for the AWS Secrets Manager Provider TestDoubles matching/capture skill fix. + +## Scope + +New evals added in `skills/compono-evals/evals.json`: + +- 28 — `IConfigurationBuilder` argument-filtered verification +- 29 — `IAmazonSecretsManager` argument-matched configuration +- 30 — adversarial NSubstitute vocabulary trap +- 31 — true callback/capture boundary + +## Runner + +Executed with Pi print mode against the repo skill source: + +```bash +pi --print --no-context-files --no-skills --skill --thinking minimal "" +``` + +Claude Code was attempted first but was unavailable because the local account had hit its weekly limit. The attempted outputs contained only the limit message and were discarded. + +## Baseline + +This PR improves an existing skill, so per the skill-creator workflow each +`with_skill` run is paired against an `old_skill` baseline, and both +configurations are launched together in the same batch rather than at +different times: the full `skills/compono` directory (`SKILL.md` + +`references/`, not just `testdoubles.md`) was snapshotted at `644a5ad` +(the commit before this PR's fix, `5139507`/`caae88c`), and all 8 runs +(evals 28–31 x with_skill/old_skill) were launched together against the +current skill and the `644a5ad` snapshot respectively, using the same +runner and prompts. Outputs are in `outputs/old_skill/`. + +An earlier version of this benchmark *described* the baseline as covering +only `testdoubles.md` and ran `old_skill` in a separate pass after +`with_skill` had already been recorded — flagged in review as not a true +paired run and, separately, as under-scoped versus the fix (`5139507` also +changed `SKILL.md`, adding a guardrail section). The underlying snapshot +had in fact already captured the full directory including the pre-fix +`SKILL.md`, so only the description was wrong; the under-scoping concern +itself didn't hold, +but the not-launched-together concern was valid — this rerun replaces that +version. + +## Results + +| Eval | with_skill | old_skill | Evidence | +|---|---:|---:|---| +| 28 | Pass | **Fail** | `with_skill` uses `configurationBuilder.Configure().Add(Match.Any()).Returns(configurationBuilder)` and `configurationBuilder.Verify().Add(Match.Is(...)).Once()`, maps `Arg.Is` to `Match.Is`, no fake recommended. `old_skill` claims the argument predicate can't be expressed at all, uses zero-arg `Configure().Add()`/`Verify().Add().Once()`, and says asserting the argument type still requires keeping the test on NSubstitute. | +| 29 | Pass | **Fail** | `with_skill` uses `secretsManager.Configure().GetSecretValueAsync(Match.Is(...), Match.Any()).Returns(...)`; maps `Arg.Is`/`Arg.Any` directly to `Match.Is`/`Match.Any`. `old_skill` drops the argument matcher entirely, uses parameterless `Configure().GetSecretValueAsync().Returns(...)`, and states outright that `Compono.TestDoubles is argument-independent`. | +| 30 | Pass | **Fail** | `with_skill` maps `Arg.Is`/`Arg.Any`/`Received(1)`/`Received(2)`/`DidNotReceive` to `Match.Is`/`Match.Any`/`Verify().Once()`/`Exactly(2)`/`Never()` and rejects recording fakes solely for NSubstitute vocabulary. `old_skill` says `Arg.Any`/`Arg.Is` "usually disappears" and lists argument-specific returns/verification as behavior Compono.TestDoubles does not provide. | +| 31 | Pass | **Fail** | `with_skill` distinguishes ordinary matching/filtering from invocation-aware callback/capture behavior and recommends a project-local fake for the unsupported callback boundary. `old_skill` reaches the same fake recommendation but for the wrong reason — it claims there are "no argument matchers"/"no argument-aware behavior" at all, contradicting the expectation that ordinary matching is supported. | + +Focused new-regression pass rate: **with_skill 4/4 (100%)**, **old_skill 0/4 (0%)** — +the fix demonstrably corrects all four scenarios the pre-fix skill got wrong. + +## Existing eval behavior + +The existing eval definitions (1–27) were left unchanged; only evals 28–31 were appended. A full historical with/without-skill benchmark could not be rerun in this environment because Claude Code returned the local weekly-limit message. No comparable old-eval regression benchmark is recorded here. diff --git a/skills/compono-evals/benchmarks/2026-08-27/outputs/eval-28-with-skill.md b/skills/compono-evals/benchmarks/2026-08-27/outputs/eval-28-with-skill.md new file mode 100644 index 0000000..57de7e0 --- /dev/null +++ b/skills/compono-evals/benchmarks/2026-08-27/outputs/eval-28-with-skill.md @@ -0,0 +1,36 @@ +Use the generated double’s `Configure()`/`Verify()` surface, not `Received`/`Arg`. + +```csharp +[Theory] +[Compose] +public void Adds_secrets_manager_configuration_source(IConfigurationBuilder configurationBuilder) +{ + // IConfigurationBuilder.Add returns IConfigurationBuilder, so configure the + // generated double to return itself when the extension method calls Add. + configurationBuilder.Configure() + .Add(Match.Any()) + .Returns(configurationBuilder); + + configurationBuilder.AddSecretsManager(/* test args */); + + configurationBuilder.Verify() + .Add(Match.Is( + source => source is SecretsManagerConfigurationSource)) + .Once(); +} +``` + +If the `IConfigurationBuilder` is also injected into a composed SUT/collaborator, mark it shared: + +```csharp +public void Test([Shared] IConfigurationBuilder configurationBuilder, SomeService sut) +``` + +But for a direct extension-method test, the key migration is: + +```csharp +configurationBuilder.Verify() + .Add(Match.Is( + source => source is SecretsManagerConfigurationSource)) + .Once(); +``` diff --git a/skills/compono-evals/benchmarks/2026-08-27/outputs/eval-29-with-skill.md b/skills/compono-evals/benchmarks/2026-08-27/outputs/eval-29-with-skill.md new file mode 100644 index 0000000..4c64a5e --- /dev/null +++ b/skills/compono-evals/benchmarks/2026-08-27/outputs/eval-29-with-skill.md @@ -0,0 +1,31 @@ +Use the generated double’s `Configure()` surface and Compono matchers: + +```csharp +secretsManager.Configure() + .GetSecretValueAsync( + Match.Is(request => request.SecretId == secretName), + Match.Any()) + .Returns(Task.FromResult(response)); +``` + +Full shape in a Compono setup: + +```csharp +var composer = Composer.Create(builder => builder.UseGeneratedTestDoubles()); + +var secretsManager = composer.Create(); + +secretsManager.Configure() + .GetSecretValueAsync( + Match.Is(request => request.SecretId == secretName), + Match.Any()) + .Returns(Task.FromResult(response)); +``` + +If this double is injected into a composed SUT/test row, make sure the same `IAmazonSecretsManager` instance is shared with the SUT, e.g. with `[Shared]` in `Compono.XunitV3`/`Compono.TUnit`. + +Mapping from the old NSubstitute call: + +- `Arg.Is(...)` → `Match.Is(...)` +- `Arg.Any()` → `Match.Any()` +- `.Returns(Task.FromResult(response))` stays the terminal configuration call. diff --git a/skills/compono-evals/benchmarks/2026-08-27/outputs/eval-30-with-skill.md b/skills/compono-evals/benchmarks/2026-08-27/outputs/eval-30-with-skill.md new file mode 100644 index 0000000..14ad4e4 --- /dev/null +++ b/skills/compono-evals/benchmarks/2026-08-27/outputs/eval-30-with-skill.md @@ -0,0 +1,38 @@ +No — not solely because the old tests used NSubstitute APIs. + +For eligible non-overloaded interface members, migrate the NSubstitute vocabulary directly to `Compono.TestDoubles` generated-double APIs: + +```csharp +Arg.Is(predicate) -> Match.Is(predicate) +Arg.Any() -> Match.Any() +Received(1).Member(...) -> Verify().Member(...).Once() +Received(2).Member(...) -> Verify().Member(...).Exactly(2) +DidNotReceive().Member(...) -> Verify().Member(...).Never() +``` + +Examples for the AWS Secrets Manager Provider shapes: + +```csharp +secretsManager.Configure() + .GetSecretValueAsync( + Match.Is(r => r.SecretId == secretName), + Match.Any()) + .Returns(Task.FromResult(response)); +``` + +```csharp +configurationBuilder.Verify() + .Add(Match.Is( + source => source is SecretsManagerConfigurationSource)) + .Once(); +``` + +Use hand-written recording fakes only if the test genuinely needs something outside current generated doubles, such as: + +- true argument capture for later arbitrary inspection, +- invocation-aware callbacks/side effects, +- call-order verification, +- sequential/call-count-based responses, +- unsupported member shapes. + +If the members are eligible non-overloaded interface members and the project has `Compono.TestDoubles`, `UseGeneratedTestDoubles()`, and `true`, generated doubles are the intended migration path. Also remember to use `[Shared]` when the test needs to configure/verify the same double instance that is injected into the composed SUT. diff --git a/skills/compono-evals/benchmarks/2026-08-27/outputs/eval-31-with-skill.md b/skills/compono-evals/benchmarks/2026-08-27/outputs/eval-31-with-skill.md new file mode 100644 index 0000000..71fd786 --- /dev/null +++ b/skills/compono-evals/benchmarks/2026-08-27/outputs/eval-31-with-skill.md @@ -0,0 +1,26 @@ +Use a local fake (or keep NSubstitute for that seam if the project allows it). + +`Match.Any()` / `Match.Is()` only express **argument matching** for configuration/verification, e.g. “when called with any delegate and any token, return this fixed value” or “verify it was called once with a delegate matching this predicate.” + +They do **not** support invocation-aware callbacks such as: + +- inspect the actual invocation as a call object +- invoke the actual `RequestHandlerDelegate` argument +- run side effects before/after that invocation +- return the delegate’s actual result + +So this NSubstitute shape: + +```csharp +pipeline.InvokeAsync( + Arg.Any(), + Arg.Any()) +.Returns(call => +{ + // use actual delegate argument +}); +``` + +doesn’t translate to `Compono.TestDoubles` matchers. + +A small local fake is the right migration boundary here, e.g. a fake pipeline that records before/after, calls the supplied `RequestHandlerDelegate`, and returns its `Task`/result. Use TestDoubles for plain matched returns and filtered verification; use a fake when the behavior depends on executing the invocation itself. diff --git a/skills/compono-evals/benchmarks/2026-08-27/outputs/old_skill/eval-28-old-skill.md b/skills/compono-evals/benchmarks/2026-08-27/outputs/old_skill/eval-28-old-skill.md new file mode 100644 index 0000000..171d48e --- /dev/null +++ b/skills/compono-evals/benchmarks/2026-08-27/outputs/old_skill/eval-28-old-skill.md @@ -0,0 +1,38 @@ +With generated Compono test doubles, don’t use `Received`/`Arg.Is`. Verification is argument-independent, and because `IConfigurationBuilder.Add` is non-overloaded, the generated verification member is parameterless: + +```csharp +[Theory] +[Compose] +public void Adds_secrets_manager_configuration_source( + [Shared] IConfigurationBuilder configurationBuilder) +{ + configurationBuilder.AddSecretsManager(); + + configurationBuilder.Verify().Add().Once(); +} +``` + +If the extension/method under test uses the return value of `IConfigurationBuilder.Add(...)`, configure the fluent return first: + +```csharp +[Theory] +[Compose] +public void Adds_secrets_manager_configuration_source( + [Shared] IConfigurationBuilder configurationBuilder) +{ + configurationBuilder.Configure().Add().Returns(configurationBuilder); + + configurationBuilder.AddSecretsManager(); + + configurationBuilder.Verify().Add().Once(); +} +``` + +Key migration point: `Compono.TestDoubles` cannot express the old argument predicate: + +```csharp +Arg.Is(source => + source is SecretsManagerConfigurationSource) +``` + +So `Verify().Add().Once()` verifies that `Add` was called once, but not which `IConfigurationSource` instance/type was passed. If asserting the argument type is essential, keep that particular test on NSubstitute or test through a real/recording `IConfigurationBuilder` instead. diff --git a/skills/compono-evals/benchmarks/2026-08-27/outputs/old_skill/eval-29-old-skill.md b/skills/compono-evals/benchmarks/2026-08-27/outputs/old_skill/eval-29-old-skill.md new file mode 100644 index 0000000..9eddbcb --- /dev/null +++ b/skills/compono-evals/benchmarks/2026-08-27/outputs/old_skill/eval-29-old-skill.md @@ -0,0 +1,45 @@ +Use the generated-double `Configure()` surface, not NSubstitute argument matchers: + +```csharp +secretsManager + .Configure() + .GetSecretValueAsync() + .Returns(Task.FromResult(response)); +``` + +Current Compono setup should look like this: + +```csharp +public sealed class GeneratedTestDoubleProfile : ICompositionProfile +{ + public void Configure(CompositionBuilder builder) => + builder.UseGeneratedTestDoubles(); +} +``` + +And the test needs the shared generated double so the configured instance is the same one injected into the SUT: + +```csharp +[Theory] +[Compose] +public async Task Reads_secret( + [Shared] IAmazonSecretsManager secretsManager, + MySecretsProvider sut) +{ + var secretName = "my-secret"; + + var response = new GetSecretValueResponse + { + SecretString = """{"key":"value"}""" + }; + + secretsManager + .Configure() + .GetSecretValueAsync() + .Returns(Task.FromResult(response)); + + // act/assert... +} +``` + +Important difference from NSubstitute: `Compono.TestDoubles` is argument-independent. This setup returns `response` for every `GetSecretValueAsync(...)` call. Since `GetSecretValueAsync` is non-overloaded here, the generated configuration member takes no arguments. diff --git a/skills/compono-evals/benchmarks/2026-08-27/outputs/old_skill/eval-30-old-skill.md b/skills/compono-evals/benchmarks/2026-08-27/outputs/old_skill/eval-30-old-skill.md new file mode 100644 index 0000000..b543692 --- /dev/null +++ b/skills/compono-evals/benchmarks/2026-08-27/outputs/old_skill/eval-30-old-skill.md @@ -0,0 +1,41 @@ +No — not just because the old tests used NSubstitute vocabulary. + +If the migrated members are eligible, non-overloaded interface members and the project intentionally uses `Compono.TestDoubles`, prefer generated doubles: + +- `Arg.Any` / `Arg.Is` usually disappears because generated doubles are argument-independent. +- `Received(1)` → `double.Verify().Member().Once()` +- `Received(2)` → `double.Verify().Member().Exactly(2)` +- `DidNotReceive()` → `double.Verify().Member().Never()` +- return setup → `double.Configure().Member().Returns(...)` +- exception setup → `double.Configure().Member().Throws(...)` + +Use `[Shared]` when the test needs to configure/verify the same double that was injected into the SUT: + +```csharp +[Theory] +[Compose] +public async Task Example( + [Shared] IAmazonSecretsManager secretsManager, + AwsSecretsManagerProvider sut) +{ + secretsManager.Configure() + .GetSecretValueAsync() + .Returns(Task.FromResult(response)); + + await sut.LoadAsync(); + + secretsManager.Verify() + .GetSecretValueAsync() + .Once(); +} +``` + +Only write recording fakes if the test truly needs behavior `Compono.TestDoubles` does not provide, such as: + +- argument-specific returns, +- argument-specific verification, +- call-order assertions, +- capturing arguments for assertions, +- unsupported interface/member shapes. + +For simple migrated `Arg.Any`, `Received(n)`, and `DidNotReceive` against eligible non-overloaded members, generated `Configure()`/`Verify()` is the intended replacement, not hand-written fakes. diff --git a/skills/compono-evals/benchmarks/2026-08-27/outputs/old_skill/eval-31-old-skill.md b/skills/compono-evals/benchmarks/2026-08-27/outputs/old_skill/eval-31-old-skill.md new file mode 100644 index 0000000..204cbf1 --- /dev/null +++ b/skills/compono-evals/benchmarks/2026-08-27/outputs/old_skill/eval-31-old-skill.md @@ -0,0 +1,48 @@ +No — don’t try to express that with `Match.Any()` / `Match.Is()` in `Compono.TestDoubles`. + +`Compono.TestDoubles` is intentionally not a general mocking framework: + +- no argument matchers +- no argument-aware behavior +- no callback/`Returns(callInfo => ...)` equivalent +- configuration is per member/overload, not per argument value +- it cannot invoke one of the actual arguments and return that result + +For this shape: + +```csharp +pipeline.InvokeAsync( + Arg.Any(), + Arg.Any()) +``` + +where the fake must: + +1. capture the real `RequestHandlerDelegate` +2. run side effects before/after +3. invoke that delegate +4. return the delegate’s result + +you should keep a local fake/stub implementation, or keep using `Compono.NSubstitute` for that specific interface if you still want mocking-framework behavior there. + +A hand fake is usually clearest: + +```csharp +private sealed class RecordingPipeline : IPipeline +{ + public bool BeforeCalled { get; private set; } + public bool AfterCalled { get; private set; } + + public async Task InvokeAsync( + RequestHandlerDelegate next, + CancellationToken cancellationToken) + { + BeforeCalled = true; + var result = await next(); + AfterCalled = true; + return result; + } +} +``` + +Use generated TestDoubles for simple “return this” / “throw this” collaborators. For callback-style middleware/pipeline behavior, a local fake is the right migration target. diff --git a/skills/compono-evals/evals.json b/skills/compono-evals/evals.json index 86bc828..60c3f9a 100644 --- a/skills/compono-evals/evals.json +++ b/skills/compono-evals/evals.json @@ -315,6 +315,56 @@ "Does not imply per-profile/scoped constructor selection is supported anywhere", "Does not conflate CMP0033 and CMP0034 as the same problem" ] + }, + { + "id": 28, + "category": "migration", + "prompt": "Migrate this AWS Secrets Manager Provider xUnit v3 test from AutoFixture/NSubstitute to Compono.XunitV3 + Compono.TestDoubles. The project references Compono, Compono.XunitV3, and Compono.TestDoubles, has true, and the profile calls UseGeneratedTestDoubles(). The old assertion is: configurationBuilder.Received(1).Add(Arg.Is(source => source is SecretsManagerConfigurationSource)); IConfigurationBuilder.Add is a non-overloaded interface member in this project. What should the Compono version look like?", + "expected_output": "Uses the generated TestDoubles verification surface: configurationBuilder.Verify().Add(Match.Is(source => source is SecretsManagerConfigurationSource)).Once(); Explains Arg.Is maps to Match.Is and Received(1) maps to Verify().Once() for an eligible member. Does not introduce a hand-written recording fake merely because Arg.Is appears.", + "files": [], + "expectations": [ + "Uses configurationBuilder.Verify().Add(Match.Is(...)).Once() for the migrated assertion", + "Correctly maps Arg.Is(predicate) to Match.Is(predicate) and Received(1) to Verify().Once()", + "Does not recommend or introduce a hand-written recording fake solely because the original NSubstitute assertion used Arg.Is" + ] + }, + { + "id": 29, + "category": "migration", + "prompt": "Migrate this AWS Secrets Manager Provider setup from NSubstitute to Compono.TestDoubles. The project has Compono.TestDoubles configured and IAmazonSecretsManager is resolved as a generated double. Old code: secretsManager.GetSecretValueAsync(Arg.Is(request => request.SecretId == secretName), Arg.Any()).Returns(Task.FromResult(response)); GetSecretValueAsync is a non-overloaded interface member for this usage. Show the current Compono setup.", + "expected_output": "Uses secretsManager.Configure().GetSecretValueAsync(Match.Is(request => request.SecretId == secretName), Match.Any()).Returns(Task.FromResult(response)); Does not claim TestDoubles lacks argument matching, and does not fall back to NSubstitute or a fake for this ordinary argument-matched configuration.", + "files": [], + "expectations": [ + "Uses secretsManager.Configure().GetSecretValueAsync(Match.Is(...), Match.Any()).Returns(...) for the migrated setup", + "Correctly maps Arg.Any() to Match.Any() and Arg.Is(predicate) to Match.Is(predicate)", + "Does not claim Compono.TestDoubles lacks argument matching or require a hand-written fake/NSubstitute for this eligible member" + ] + }, + { + "id": 30, + "category": "behavioral-correctness", + "prompt": "I'm reviewing a Compono.TestDoubles migration in AWS Secrets Manager Provider. The old tests are full of NSubstitute vocabulary: Arg.Is, Arg.Any, Received(1), Received(2), and DidNotReceive. The project intentionally removed NSubstitute and Compono.NSubstitute; it now references Compono.XunitV3 and Compono.TestDoubles with generated doubles enabled. Should I write recording fakes for IAmazonSecretsManager, IConfigurationBuilder, and ILoggerFactory because the old tests used those NSubstitute APIs? Assume the members being migrated are eligible non-overloaded interface members.", + "expected_output": "Says no: NSubstitute vocabulary alone is not evidence a fake is required. Translates Arg.Is -> Match.Is, Arg.Any -> Match.Any, Received(1) -> Verify().Member(...).Once(), Received(n) -> Verify().Member(...).Exactly(n), and DidNotReceive() -> Verify().Member(...).Never() for eligible TestDoubles members. Recommends local fakes only for actual unsupported boundaries such as capture/callback/call order, not for ordinary matching/filtering.", + "files": [], + "expectations": [ + "Explicitly rejects writing recording fakes solely because the original tests use Arg.Is/Arg.Any/Received/DidNotReceive", + "Maps Arg.Is to Match.Is and Arg.Any to Match.Any", + "Maps Received(1) to Verify().Once(), Received(n) to Verify().Exactly(n), and DidNotReceive() to Verify().Never() where the member shape is eligible", + "Distinguishes ordinary matching/filtering from actual unsupported boundaries such as capture, callbacks, and call-order verification" + ] + }, + { + "id": 31, + "category": "behavioral-correctness", + "prompt": "In a Compono.TestDoubles migration, one old NSubstitute setup uses a callback-style response: when pipeline.InvokeAsync(Arg.Any(), Arg.Any()) is called, the substitute must invoke the actual RequestHandlerDelegate argument, record side effects around that invocation, and return the delegate's result. Can I express this with Match.Any()/Match.Is() in TestDoubles, or should I keep a local fake?", + "expected_output": "Explains the boundary: ordinary argument matching/filtering is supported by Match.Any/Match.Is and Verify(...).Once/Never/Exactly, but this scenario is stronger invocation-aware callback/capture behavior. Match only decides whether an argument matches; it does not expose the actual invocation to Returns, invoke delegate arguments, or run callback side effects. Recommends following existing roadmap/project-local-fake guidance for that unsupported boundary rather than claiming Match solves it.", + "files": [], + "expectations": [ + "States that ordinary matching and argument-filtered verification are supported by Compono.TestDoubles", + "Clearly says the described callback/capture scenario is a different, stronger capability than matching", + "Does not claim Match can inspect the actual invocation inside Returns, invoke delegate arguments, or implement callback side effects", + "Recommends a project-local fake or existing roadmap guidance for the unsupported invocation-aware callback boundary" + ] } ] -} \ No newline at end of file +} diff --git a/skills/compono/SKILL.md b/skills/compono/SKILL.md index 862f888..38e8827 100644 --- a/skills/compono/SKILL.md +++ b/skills/compono/SKILL.md @@ -74,6 +74,29 @@ user to make test-by-test, not something to do as a drive-by. ## Default workflow +### High-priority `Compono.TestDoubles` matching guardrail + +If the project references `Compono.TestDoubles`, calls +`UseGeneratedTestDoubles()`, mentions generated doubles, or asks to migrate +NSubstitute `Arg.Is`/`Arg.Any`/`Received`/`DidNotReceive` usage to current +TestDoubles, **read `references/testdoubles.md` before answering**. Do not +answer from memory: older Compono guidance said generated doubles had no +argument matching, but that is stale. Current TestDoubles supports +`Configure()`, `Verify()`, literal equality matching, `Match.Any()`, +`Match.Is(predicate)`, argument-filtered `Never()`/`Once()`/`Exactly(n)`, +and multi-entry argument-distinguished response configuration for eligible +member shapes. Translate NSubstitute vocabulary directly where eligible: +`Arg.Is` → `Match.Is`, `Arg.Any()` → `Match.Any()`, +`Received(1)` → `Verify().Member(...).Once()`, `Received(n)` → +`Verify().Member(...).Exactly(n)`, and `DidNotReceive()` → +`Verify().Member(...).Never()`. Never invent non-existent TestDoubles APIs +such as `CallsTo(...)`, `ReceivedCalls()`, or `[ComponoTest]`, and never +recommend a hand-written recording fake solely because the old test used +NSubstitute argument matchers. True argument capture, invocation-aware +callback responses/side effects, and call-order verification are different +capabilities; use project-local fake/roadmap guidance for those boundaries +instead of claiming `Match` solves them. + 1. **Detect** — run the table above. Know which packages are actually installed before recommending any API from them. 2. **Inspect** the type under test and its collaborators — concrete class @@ -96,11 +119,21 @@ user to make test-by-test, not something to do as a drive-by. performance win; ordinary composition is already cheap. - Interface/abstract-class/delegate needs a real test double → `Compono.NSubstitute`'s `UseNSubstitute()`, not a hand-rolled stub, - if that package is referenced. An **interface** leaf that only needs - configured returns/exceptions (no call verification, no argument - matchers) and must survive `PublishAot` → `Compono.TestDoubles`'s - `UseGeneratedTestDoubles()` instead, if that package is referenced and - the compile-time opt-in is set — see `references/testdoubles.md`. + if that package is referenced. An **interface** leaf that should be + source-generated/AOT-safe → `Compono.TestDoubles`'s + `UseGeneratedTestDoubles()`, if that package is referenced and the + compile-time opt-in is set. Current generated doubles support + `Configure()`, `Verify()`, literal equality matching, `Match.Any()`, + `Match.Is(predicate)`, argument-filtered `Never()`/`Once()`/ + `Exactly(n)`, and multi-entry argument-distinguished response + configuration for eligible member shapes. Do not mistake NSubstitute + vocabulary (`Arg.Is`, `Arg.Any`, `Received`, `DidNotReceive`) for a + reason to invent a hand-written recording fake; translate it to the + generated-double surface where the member shape is eligible. True + argument capture, invocation-aware callback responses/side effects, + call-order verification, classes, delegates, and other explicitly + unsupported shapes remain outside current `Compono.TestDoubles` + support — see `references/testdoubles.md`. - A test deliberately needs to exercise the real HTTP client pipeline (real `HttpClient` → `TestHttpHandler` → configured response) rather than substitute an application-level interface away → diff --git a/skills/compono/references/testdoubles.md b/skills/compono/references/testdoubles.md index d2dc7da..a3e8403 100644 --- a/skills/compono/references/testdoubles.md +++ b/skills/compono/references/testdoubles.md @@ -37,11 +37,21 @@ service.Repository.Configure().CountAsync().Returns(Task.FromResult(4)); `composer.Create()`/`CreateMany()` call site, a `[Compose]` theory/ test method parameter, or a `[Composable]` declaration all feed the same closure walk). -- **`.Returns(...)`/`.Throws(...)`** per member. Argument-independent — - there is no `Arg.Any()`/argument-matcher equivalent; configuration - applies to every call to that member regardless of arguments. Last - configuration wins: calling `.Returns(...)` after an earlier - `.Throws(...)` on the same member clears the exception (and vice versa). +- **`.Returns(...)`/`.Throws(...)`** per member configuration. + Zero-argument `Configure().Member()` configures the member regardless of + arguments. For eligible parameterized members, `Configure().Member(...)` + accepts `Match` arguments: literal values match by equality, + `Match.Any()` matches anything, and `Match.Is(predicate)` matches + by predicate. +- **`Verify()`** — parallel to and independent from `Configure()`, returning + a verifier surface for `Never()`/`Once()`/`Exactly(n)`. Zero-argument + `Verify().Member()` counts every call to that member. For eligible + parameterized members, `Verify().Member(...)` uses the same `Match` + argument shape to perform argument-filtered verification. +- **Multi-entry argument-distinguished configuration** — for matching- + eligible members, multiple `Configure().Member(...)` calls can coexist. + Dispatch uses the most recently registered matching entry; precedence is + registration order, not matcher-specificity ranking. - **Full base-interface closure.** If `IRepository : IClock`, the generated double implements `IClock.UtcNow` too, configurable via `repository.Configure().UtcNow().Returns(...)` — not just `IRepository`'s @@ -51,25 +61,96 @@ service.Repository.Configure().CountAsync().Returns(Task.FromResult(4)); known collection shapes return their deterministic default (empty collections, never `null`). `Task`/`ValueTask` recurse into `T` — `Task` is fine, but `Task` (a non-nullable reference `T`) - has no deterministic default for its result and hits the same diagnostic - as a bare non-nullable reference return. A member with **no** - deterministic default — a non-nullable reference return (`string`, a - non-nullable class), or a `Task`/`ValueTask` wrapping one — is a - compile-time diagnostic instead; the generator never emits `null` for a - non-nullable-annotated return. + may require explicit configuration rather than a default. See + "Configuration-required members" below. + +## Argument matching and filtered verification + +Do not conflate argument matching with argument capture. Current +`Compono.TestDoubles` supports ordinary matcher-based configuration and +verification for eligible members; it does not expose an arbitrary call log +or invocation callback API. + +### NSubstitute migration mapping + +When migrating from NSubstitute, do not introduce a hand-written recording +fake merely because the old test uses `Arg.Is`, `Arg.Any`, `Received`, or +`DidNotReceive`. For an eligible generated-double member, translate the +concepts directly: + +| NSubstitute | Compono.TestDoubles | +|---|---| +| `Arg.Is(predicate)` | `Match.Is(predicate)` | +| `Arg.Any()` | `Match.Any()` | +| literal argument | literal argument (equality match) | +| `Received(1).Member(...)` | `Verify().Member(...).Once()` | +| `Received(n).Member(...)` | `Verify().Member(...).Exactly(n)` | +| `DidNotReceive().Member(...)` | `Verify().Member(...).Never()` | + +Real AWS Secrets Manager Provider migration shapes: + +```csharp +configurationBuilder.Verify() + .Add(Match.Is(source => source is SecretsManagerConfigurationSource)) + .Once(); +``` + +```csharp +secretsManager.Configure() + .GetSecretValueAsync( + Match.Is(request => request.SecretId == secretName), + Match.Any()) + .Returns(Task.FromResult(response)); +``` + +### Eligibility boundary + +Argument-aware `Configure().Member(...)`/`Verify().Member(...)` is generated +for a member only when the member is eligible: it is not overloaded, its +real parameters do not reference the member's own open generic type +parameter, its real parameters are usable as generic type arguments, the +derived generated field names do not collide, and its generated extension +would not be hidden by an inherited `object` member. If a member is scoped +out of argument-awareness, keep using the existing argument-independent +surface (`Configure().Member()` / `Verify().Member()`) or choose another +test seam/provider when the test genuinely needs argument distinction. + +Overloaded members keep the discriminator-only shape described below: their +arguments select the overload at compile time; they are not matchers. + +## Multiple response configurations per member + +For matching-eligible members, each `Configure().Member(...)` call appends a +new response configuration. Dispatch walks matching entries from newest to +oldest and uses the first match: + +```csharp +repository.Configure() + .Withdraw(Match.Any(), Match.Any(), Match.Any()) + .Returns(false); +repository.Configure() + .Withdraw("acct-1", Match.Any(), Match.Any()) + .Returns(true); +``` + +`Withdraw("acct-1", ...)` returns `true`; other accounts return `false`. +There is no matcher-specificity ranking — if two entries both match, the +one configured later wins. + +This is still not sequential/call-count-based responses. There is no +"return X on the first call, Y on the second" API. ## Overloaded members (v2) -An overloaded interface member now gets its own per-overload `Configure()` +An overloaded interface member gets its own per-overload `Configure()` surface instead of an all-or-nothing rejection (see `docs/adr/0044-compono-testdoubles-v2-overloads-generics-verification.md`) — the generated configuration extension for an overloaded member takes the same real parameter types the interface overload declares, purely so -ordinary C# overload resolution picks the right one (the values themselves -are still discarded, same as the non-overloaded, zero-argument case). -`Verify()` reuses this same per-overload surface - `Verify().Speak("hi")` -selects the same overload-specific counter `Configure().Speak("hi")` -would: +ordinary C# overload resolution picks the right one. The values themselves +are still discarded and are **not** argument matchers. `Verify()` reuses +this same per-overload surface - `Verify().Speak("hi")` selects the same +overload-specific counter `Configure().Speak("hi")` would: ```csharp public interface IResponseBuilder @@ -133,23 +214,25 @@ argument is needed at the call site whenever ordinary overload-resolution betterness rules wouldn't otherwise pick that overload (same as a real call to the interface member itself). -**Still unsupported:** a generic method whose return type depends on its -own type parameter (`T Get()`) - no constructible fallback body, whole -interface falls back (`CMP0031`). **Any** type parameter used as `T?` in a -parameter is diagnosed and excluded too (`CMP0026`) - constrained or -unconstrained, regardless of which constraint; correctly modeling exactly -when (and with which keyword) a constraint restatement is required isn't -attempted. +A generic method whose return type depends directly on its own type +parameter (`T Get()`, `Task GetAsync()`, and supported nullable +variants) can have per-closed-`T` configuration. Argument matching is +available for that shape only when the member's parameters are otherwise +matching-eligible and do not use the method's own open type parameter. + +Still unsupported: unsupported generic-return shapes beyond the documented +per-closed-`T` cases, value-type-constrained `T?` (`System.Nullable`), +and parameter shapes the generator cannot represent without reflection or +boxing. See `diagnostics.md` before guessing a workaround. -## Call verification (v2) +## Call verification (v2+) `Verify()` — parallel to and independent from `Configure()`, returning a -distinct wrapper so the two never collide — asserts how many times a -member was actually called -(`docs/adr/0044-compono-testdoubles-v2-overloads-generics-verification.md` -Requirement 3). `Never()`/`Once()`/`Exactly(n)` only, argument-independent -(same as `Configure()`), reusing the same per-overload discriminator -`Configure()` does: +distinct wrapper so the two never collide — asserts how many times a member +was actually called. `Never()`/`Once()`/`Exactly(n)` are the terminal +count assertions. For argument-aware members, filtering happens in the +generated `Verify().Member(...)` extension before the terminal +`CallVerifier` is returned: ```csharp repository.Configure().CountAsync().Returns(Task.FromResult(5)); @@ -158,6 +241,12 @@ repository.Verify().CountAsync().Once(); repository.Verify().Save().Once(); ``` +```csharp +repository.Verify() + .Save(Match.Is(order => order.Id == expectedId)) + .Once(); +``` + A failing assertion throws `Compono.TestDoubleVerificationException` (a plain exception, not a framework assertion type). A call counts whether it hits configured, default, or thrown behavior. @@ -190,18 +279,33 @@ This applies identically to sync/async/property members and to a fluent self-returning member (`IResponseBuilder`-shaped) — none of those get special-cased, all follow the same rule. -## The #1 AutoFixture/NSubstitute-habit trap: not a general mocking framework - -There are **no** argument matchers, **no** argument-aware call recording -(every count is per-member, not per-argument-combination), and **no** -call-order verification. If a test needs different return values for -different arguments, or needs to assert *when* relative to other calls a -member ran, `Compono.TestDoubles` cannot do it — use -`Compono.NSubstitute`'s `UseNSubstitute()` for that interface instead (the -two providers can coexist; registration order decides which one resolves -first, see below). Don't try to work around the gap by polling state or -inventing a callback-shaped member on the interface just to observe a -call — that's fighting the framework, not using it. +## The #1 AutoFixture/NSubstitute-habit trap: matching is not capture + +`Compono.TestDoubles` is not a general-purpose mocking framework, but it +does support argument matching and argument-filtered verification for the +eligible member shapes above. The remaining boundary is stronger behavior +that needs access to the actual invocation as a first-class value: + +- true argument capture for later arbitrary inspection outside a generated + `Verify().Member(Match...)` count assertion; +- invocation-aware callback responses (`Returns(call => ...)`, + `Returns(Func)`, or "invoke this delegate argument and use + its result"); +- callback side effects based on the actual invocation; +- call-order verification; +- sequential/call-count-based responses; +- strict mode, partial substitutes, recursive auto-configuration; +- classes, delegates, indexers, events, and other unsupported shapes listed + below. + +If a test only needs "this member was called once with an argument matching +this predicate," use `Verify().Member(Match.Is(...)).Once()`. If it +needs to store every argument for arbitrary later inspection, run code from +a callback, or invoke a delegate argument, that is a different capability; +use an existing project-local fake or `Compono.NSubstitute` where the +project intentionally keeps that dependency, and treat any real +`Compono.NSubstitute`-can/`Compono.TestDoubles`-cannot case as roadmap +evidence under ADR-0042 Amendment 2. ## Unsupported shapes are compile-time diagnostics, not silent gaps @@ -214,29 +318,28 @@ runtime `CompositionException` if no provider handles it, not a `CMP002x` diagnostic). For an eligible **interface**, indexers, events, a genuinely unimplemented -static abstract member, a generic method whose return type depends on its -own type parameter, a generic type parameter used as `T?` (constrained or -not), and a handful of narrower shapes (set-only properties, +static abstract member, unsupported generic-method return/parameter shapes, +and a handful of narrower shapes (set-only properties, pointer/function-pointer parameters or returns, ref-like returns) still reject the **whole interface** at compile time (`CMP0020`-`CMP0031`, informational severity — they don't fail the build): it falls back to the -ordinary runtime-provider path, same as any -interface the compile-time opt-in never reached. Overloaded members, a -`ref`/`out`/`in` parameter, and a generic method independent of its own -type parameter are narrower now (see above) — only the specific -colliding/unsupported overload loses its surface, not the whole interface. -A non-nullable-reference return with no deterministic default no longer -rejects the whole interface either (v2, see "Configuration-required -members" above) — unless it also lacks a `Configure()` surface for one of -those other reasons, in which case it still does. See `diagnostics.md` for -the full code table before guessing a fix. - -A static abstract member declared on a base interface but already -resolved by a more-derived interface's own concrete implementation (C#'s -"most specific implementation" rule — the `IAmazonS3`/`IAmazonService` -shape) is **not** a genuinely unimplemented member at all and doesn't -reject anything; only a static abstract member with no override anywhere -in the interface's hierarchy still whole-interface-rejects (ADR-0046). +ordinary runtime-provider path, same as any interface the compile-time +opt-in never reached. Overloaded members, a `ref`/`out`/`in` parameter, +and a generic method independent of its own type parameter are narrower now +(see above) — only the specific colliding/unsupported overload loses its +surface, not the whole interface. A non-nullable-reference return with no +deterministic default no longer rejects the whole interface either (v2, +see "Configuration-required members" above) — unless it also lacks a +`Configure()` surface for one of those other reasons, in which case it +still does. See `diagnostics.md` for the full code table before guessing a +fix. + +A static abstract member declared on a base interface but already resolved +by a more-derived interface's own concrete implementation (C#'s "most +specific implementation" rule — the `IAmazonS3`/`IAmazonService` shape) is +**not** a genuinely unimplemented member at all and doesn't reject anything; +only a static abstract member with no override anywhere in the interface's +hierarchy still whole-interface-rejects (ADR-0046). ## Precedence with `Compono.NSubstitute` @@ -246,15 +349,14 @@ var composer = Composer.Create(builder => builder .UseNSubstitute()); ``` -Both providers can be registered together. Registration order decides -which one resolves an interface request first — `UseGeneratedTestDoubles()` +Both providers can be registered together. Registration order decides which +one resolves an interface request first — `UseGeneratedTestDoubles()` registered before `UseNSubstitute()` means any interface the generator emitted a double for resolves to the generated double; an interface that -never got a generated double falls through to `NSubstituteProvider` -(or to composition failure if neither provider claims it). This is the -same "tried in registration order" contract every provider already -follows — no special-cased precedence logic exists between these two -specifically. +never got a generated double falls through to `NSubstituteProvider` (or to +composition failure if neither provider claims it). This is the same "tried +in registration order" contract every provider already follows — no +special-cased precedence logic exists between these two specifically. ## Combining with `[Shared]` @@ -267,7 +369,7 @@ public async Task Saves_order([Shared] IRepository repository, OrderService serv { repository.Configure().CountAsync().Returns(Task.FromResult(4)); var order = await service.PlaceAsync(6); - // repository is the exact double `service` was composed with + repository.Verify().Save(Match.Is(saved => saved.Id == order.Id)).Once(); } ``` @@ -280,12 +382,12 @@ public async Task Saves_order([Shared] IRepository repository, OrderService serv { repository.Configure().CountAsync().Returns(Task.FromResult(4)); var order = await service.PlaceAsync(6); - // repository is the exact double `service` was composed with + repository.Verify().Save(Match.Is(saved => saved.Id == order.Id)).Once(); } ``` `[Shared]` (in `Compono.XunitV3` or `Compono.TUnit`) is what lets you both -configure a double *and* have it wired into the composed system under +configure/verify a double *and* have it wired into the composed system under test — see `registrations-profiles-and-scopes.md`. Without `[Shared]`, a double-typed parameter and a double nested inside another composed type would be two different generated-double instances. diff --git a/src/Compono/CallVerifier.cs b/src/Compono/CallVerifier.cs index 558d44f..b69e602 100644 --- a/src/Compono/CallVerifier.cs +++ b/src/Compono/CallVerifier.cs @@ -1,12 +1,11 @@ namespace Compono; /// -/// Asserts how many times a generated test double's member was called, backed by -/// . Deliberately minimal - / -/// / only, no argument matchers, no call-order verification, -/// per ADR-0044 Requirement 3. +/// Performs terminal call-count assertions after any generated member and argument filtering has +/// already been applied. Deliberately minimal - // +/// only, with no call-order verification, per ADR-0044 Requirement 3. /// -/// How many times the member's dispatch body actually ran. +/// How many matching calls the generated verification surface observed. /// /// The declaring interface's display name plus member name, used to describe a verification failure. ///