Retrieval quality v2: preregistered held-out benchmark; QMD hybrid beats QMD lexical search (nDCG@10 0.71 vs 0.22)

[seathatflowsinourveins]

Scope

• What this PR changes: a new foundation blueprint, blueprints/retrieval-quality-v2/. It is a preregistered held-out retrieval-quality benchmark that answers the retrieval-quality gate's stated need for "new held-out queries and relevance judgments" (catalogs/us-equities/convergence-review.json).
  • Query set: 30 held-out research questions (10 natural-language, 10 keyword, 10 paraphrased) with graded relevance judgments over the 33 pinned corpus files. A separate author agent wrote them and ran no retrieval system. They are sealed in PREREGISTRATION.md by sha256 4d0a0eaf0f6fc48b….
  • Arms, both QMD 2.8.3 on a fresh owned index under a scratch HOME, never the host's index:
    • A: lexical search;
    • B: hybrid query, on CPU, with its three models pinned by Hugging Face revision and file sha256.
• Base: current origin/main. Lane: lane:foundation.

Results (from results-20260925.json)

Arm	nDCG@10	recall@5	MRR
A: QMD lexical search	0.2227	0.2666	0.2666
B: QMD hybrid query	0.7139	0.85	0.7283

• Gain in nDCG@10: 0.4912. The paired-bootstrap 95% CI, over 10,000 resamples with seed 20260925, is [0.3673, 0.6185].
• Decision under the frozen rule: Hybrid (Arm B) selected over BM25.
• How to read it: Arm A returned no candidates for 22 of 30 queries: every natural-language and paraphrased query, and 2 of the keyword queries. The result selects QMD hybrid query over QMD lexical search as research workers would call them. It does not select hybrid over BM25 in general. A relaxed or OR BM25 baseline is a named follow-up.

Review and repair (one round)

The independent read-only review found:

• High: PREREGISTRATION.md claimed the queries were committed before any run, which was false because workers cannot commit. It now states the true ordering evidence: in workflow run wf_eaf5338d-556, the journal records the author's result before the runner's start, and the runner re-hashed the query set before any qmd call. The file carries a dated correction note.
• Moderate: the zero-hit behaviour of Arm A was under-reported. It is now in the README and in the results decision.interpretation.
• Low: one qmd --version call runs outside the scratch environment. It is documented, and the code is left exactly as run.

The Codex cross-family review is skipped until 2026-09-30.

Local commands run

validate.yml's offline validate-job steps + evidence_manifest.py --check -> 0 failures
python -m unittest -> Ran 5591 tests, OK (skipped=613)

🤖 Generated with Claude Code

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

[seathatflowsinourveins]

Cross-family review (Codex CLI 0.155.1, gpt-6-astra, model_reasoning_effort=max, read-only sandbox), 2026-09-26. Run by the workstation coordinator against a clean checkout of main after merge. Supported findings get a repair PR (one repair round; residuals recorded).

DEFECTS FOUND

Severity	File:line	What is wrong	Concrete fix
high	blueprints/retrieval-quality-v2/run.py:262	Inherited INDEX_PATH and QMD_CONFIG_DIR override scratch isolation in QMD’s database resolver and configuration resolver. Collection/index operations can therefore mutate shared host state before the document-count check fails.	Remove inherited routing/model overrides, explicitly bind configuration and database paths to owned scratch storage, and use the same explicit index for pull.
medium	blueprints/retrieval-quality-v2/run.py:588	Failed HF revision resolution returns None, but model preparation succeeds and Arm B proceeds. Reproduced with all three revisions unresolved; this violates the preregistered fail-closed rule.	Raise RunFailure on unresolved revisions and record Arm B as not evaluated before embedding/querying.
medium	blueprints/retrieval-quality-v2/run.py:401	A mismatched docid still returns the graded repository path; an empty docid passes startswith(""). A wrong-docid response reproduced nDCG=1.0 with error=null.	Require a valid six-character docid matching the pinned content hash; reject corpus mismatches before scoring.
medium	blueprints/retrieval-quality-v2/run.py:483	Wrong-shaped JSON is mishandled: {} and [42] abort the entire arm, while null records success. The frozen protocol requires malformed responses to score zero for that query and remain in the comparison.	Validate the response and hit schemas, record malformed responses as query errors, and continue subsequent queries.
medium	blueprints/retrieval-quality-v2/run.py:1016	Pull/embed timeouts bypass the RunFailure handler. Both reproduced an aborted receipt with no decision, followed by an uncaught TypeError during README rendering.	Handle setup timeouts as Arm B not evaluated; make rendering tolerate incomplete/aborted receipts.
medium	blueprints/retrieval-quality-v2/run.py:1061	Every invocation defaults to overwriting the historical results-20260925.json, including an immediate seal failure. This destroys retained evidence and can leave the old successful README beside an aborted receipt.	Use exclusive, unique run outputs; refuse existing destinations and preserve historical results and failures.
medium	blueprints/retrieval-quality-v2/run.py:499	Successful retrieval stdout/stderr is discarded; only wrapper-derived paths and metrics survive. Arithmetic is reproducible, but native rankings, docids, and the claimed observed expansion skips lack retained upstream output required by the acceptance policy.	Retain sanitized per-query native responses, stderr, argv and exit codes, with hashes linking them to the scored results.
medium	blueprints/retrieval-quality-v2/PREREGISTRATION.md:219	The preregistration ordering claim cites a workflow journal absent from the checkout. results-20260925.json:40 merely repeats the assertion; matching hashes do not establish pre-run chronology.	Retain sanitized author-completion and runner-start events bound to the original frozen artifacts, or mark preregistration chronology unverified.