perf(M-21): add interleaved A/B comparisons

[joslat]

M-21 — trustworthy in-process A/B measurement

Adds perf ab so two flag/DI-reachable configurations can be compared in one hermetic process and
database. This is a harness-only PR; it does not optimize product code.

Pre-registered prediction

Recorded in this PR before implementation:

• identical default configurations: reads 6, writes 1, queries 9, embedding requests 1, retrieved
  items 43, and access-tracked items 25 on both sides; Recall@K/MRR 1.000; elapsed ratio CI contains
  1.0, so the verdict is no significant difference;
• Recall.MaxEntities 10 → 2: entities 10 → 2, retrieved items 43 → 35, access-tracked items
  25 → 17; reads/writes/queries/embedding requests unchanged; Recall@K approximately 0.982 and MRR
  1.000; no timing win unless the bootstrap interval's upper bound is below 1.0.

What changed

• perf ab --control <spec> --candidate <spec> parses default, Recall.Max*, and
  Recall.MinSimilarityScore configurations.
• Order is counterbalanced as AB, BA; each configuration crosses over two equivalent,
  owner-isolated fixture copies.
• Six consecutive paired iterations are one bootstrap unit, preserving observed Docker/driver
  autocorrelation.
• The generated Markdown and JSON put exact counter ranges, retrieval quality, extraction quality,
  p50 timings, candidate/control ratios, and bootstrap 95% CIs together.
• iteration total is the pre-registered timing headline. Subspan intervals are explicitly
  exploratory because they are not corrected for multiple comparisons.
• State-mutating scenarios are rejected; PERF-W-02 cannot be causally paired without reset/restore.

Why the design changed from literal ABAB

The original strict-ABAB/shared-fixture implementation failed the required null test in both
directions. PERF-R-04 updates access timestamps, so one side changed the state the next side measured,
and bootstrapping correlated adjacent ratios independently made intervals too narrow. The corrected
state-isolated, dataset-crossover, counterbalanced cluster design passed the null twice.

Results

Null experiment — default vs default

Final-form runs:

• 20260726T014053Z__ab__hermetic-S-zero
• 20260726T021504Z__ab__hermetic-S-zero

Both report PASS — no significant difference, identical structural-work ranges, and zero quality
delta. Latest headline:

Metric	Control	Candidate	Delta / ratio
Iteration p50	30.22 ms	32.24 ms	1.029
Bootstrap 95% CI	—	—	0.992–1.086
Reads / writes / queries	6 / 1 / 9	6 / 1 / 9	0 / 0 / 0
Retrieved / access-tracked	43 / 25	43 / 25	0 / 0
Recall@K / MRR	1.000 / 1.000	1.000 / 1.000	0.000 / 0.000

Positive control — Recall.MaxEntities 10 → 2

Run 20260726T014215Z__ab__hermetic-S-zero:

Metric	Control	Candidate	Change
items.entities	10	2	-8 (-80.0%)
items.retrieved	43	35	-8 (-18.6%)
access_tracking.items	25	17	-8 (-32.0%)
Reads / writes / queries	6 / 1 / 9	6 / 1 / 9	0 / 0 / 0
Embedding requests	1	1	0
Iteration p50	24.65 ms	24.79 ms	ratio 0.908
Bootstrap 95% CI	—	—	0.809–1.009: no significant difference
Recall@K	1.000	0.982	-0.018
MRR / forbidden cases	1.000 / 0	1.000 / 0	unchanged
Extraction precision/recall	1.000 / 1.000	1.000 / 1.000	unchanged

The exact counter prediction held. The timing direction was lower but not significant, so this PR
does not claim a speed win. The zero-tolerance quality report correctly flags the Recall@K tradeoff.

Validation

• New tests were run before implementation and failed to compile because the A/B command,
  configuration parser, and bootstrap did not exist; focused suite now passes 13/13.
• Unit suite: 3,403/3,403 passed.
• Integration suite: 311 passed; the same 29 Nams.* tests fail because the configured workspace is
  deprovisioned (workspace_not_provisioned), as documented before this PR.
• dotnet build AgentMemory.slnx -c Release --no-restore: 0 warnings, 0 errors.
• Standard perf regression run 20260726T021600Z__m21-regression__hermetic-S-zero: quality 1.000
  throughout and every published structural counter unchanged.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.