retrieval-quality-v2 repairs for the GPT-6 findings on #293: owned QMD routing, fail-closed checks, retained native outputs, qmd bench

[seathatflowsinourveins]

Summary

This PR repairs the GPT-6 cross-family findings on merged #293 (blueprints/retrieval-quality-v2/) and adds QMD's own qmd bench as upstream corroboration. The rerun on CPU (QMD_FORCE_CPU=1) keeps the decision: Hybrid (Arm B) selected over BM25, with both preregistered conditions holding. Source: results-20260926T024558Z.json.

Arm	Mean nDCG@10 (30 queries)
A: QMD BM25	0.2228
B: QMD hybrid	0.7281 (2026-09-25 run: 0.7139)
Gain B − A	0.5053, bootstrap 95% CI [0.3813, 0.6329] over 10,000 resamples with seed 20260925

Native qmd bench runs QMD 2.8.3's own command and scorer on the same index. Its fixture is derived one-to-one from queries.json, and it is non-gating corroboration (upstream_native_operation):

Backend	recall@5	MRR
bm25	0.2667	0.2667
vector	0.7667	0.7214
hybrid	0.8333	0.7100
full	0.8500	0.7170

An exact-identity audit found no disagreement between the native bm25 scores and this runner's Arm A rankings.

What was wrong, and the fix (GPT-6 findings F1–F8, all verified)

• F1 routing: every qmd call, including pull, is bound to owned config and database paths. INDEX_PATH and QMD_CONFIG_DIR are never inherited.
• F2: unresolved HF revisions fail closed.
• F3: each docid is validated against the pinned content hash, and a query with an unverifiable hit scores zero.
• F4: malformed JSON scores zero and the run continues, per the frozen protocol.
• F5: a pull or embed timeout means Arm B is not evaluated.
• F6: every run writes a new output file; results-20260925.json is kept as history.
• F7: sanitized per-query native qmd responses, stderr, argv and exit codes are retained with hashes (native-*.json).
• F8: the chronology claim gets a dated correction (chronology-20260926.json).
• Verification round (GPT-6 and Opus):
  • Empty qmd bench output is no longer accepted as complete; every fixture query and all 4 backends are required.
  • Rejected-hit reasons and pull/embed metadata are sanitized.
  • Interruption bookkeeping now records the call running when the signal arrived and keeps the decision's limitations.
  • The README's cache-reuse wording is marked as an inference.
  • The stub-based reproductions of Retrieval quality v2: preregistered held-out benchmark; QMD hybrid beats QMD lexical search (nDCG@10 0.71 vs 0.22) #293's findings are committed and hashed, labelled synthetic.

Review

1. GPT-6: review of Retrieval quality v2: preregistered held-out benchmark; QMD hybrid beats QMD lexical search (nDCG@10 0.71 vs 0.22) #293.
2. Opus: repair, then an independent review, a finisher and a verification.
3. GPT-6 verification pass: F1–F8 resolved; 2 new mediums found and fixed with failing-first tests; 3 lows fixed.
4. Independent headless review: checked the retained runs and hashes.
5. Tests and controls: 30 retrieval tests pass, and all 29 mutation controls fail as required.

Residuals

• run.py at its final revision has not run against the real qmd. The committed 2026-09-26 results come from the prior revision efba31d5, and this round's changes affect only acceptance of empty bench output, sanitization and interruption bookkeeping. The full comparison was not re-run.
• Bench completeness is checked structurally only. QMD's runQuery records a backend call that throws as an empty result.
• The grand-dashboard retrieval-quality-v2 row changes in the separate lane:shared dashboard PR, because the dashboard file is a shared hot file.
• The retrieval-quality gate in catalogs/us-equities/convergence-review.json is unchanged here.

Verification

• scripts/validate.py, host_receipts.py validate, component_matrix.py --check, new_host_grand_list.py --check, evidence_manifest.py --check and validate_catalogs.py: all pass on this tree.
• Full python -m unittest on this change before its final replay onto main: Ran 6071 tests, OK (skipped=616). After the replay, the related modules and every validator were re-run and pass.

SOTA sources

• QMD at the pinned commit, including qmd bench and its scorer: https://github.com/tobi/qmd/tree/facd35e01359e59d938bc9418e93fb9318addee3 (src/bench/bench.ts, src/bench/score.ts, README "Benchmarking")
• nDCG (Järvelin & Kekäläinen 2002), ACM TOIS: https://doi.org/10.1145/582415.582418

🤖 Generated with Claude Code

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.