Deflated Sharpe and PBO (CSCV) over the 6,605 retained research candidates

[seathatflowsinourveins]

Closes audit gap from the 2026-09-24 trading wiring audit (native-agent-stack-71).

I added a deflated Sharpe ratio (DSR) and a probability-of-backtest-overfitting (PBO) check to the multi-candidate research lane. I ran both over the 6,605 retained candidate records. Neither shows evidence of skill.

Where the records are. The ledger itself is not in blueprints/us-equities/research-evaluation; that folder only holds its hash (in receipt.json). The file with that exact hash (08ea5309…) is already in the repo at evidence/artifacts/gap-wave2-20260923/us-equities__portfolio-risk/raw/3/regenerated/run-1/candidate-ledger.json. It is a byte-identical regeneration of the original run. No data was refetched.

Implementation. I wrote the module from the papers rather than vendoring one. None of the candidate libraries was on this machine. eslazarev/purged-cross-validation is catalogued only as a discovery with no pin. pypbo and the R pbo package would each need a network fetch and a new dependency lock for about 150 lines of arithmetic. The module uses only the standard library, and its docstring cites Bailey and Lopez de Prado (2012, 2014) and Bailey, Borwein, Lopez de Prado and Zhu (2017). The receipt records a condition for switching to a maintained, licence-reviewed library later.

Parameters. params.json was frozen (sha256 0e371da4…) at 15:53:14Z, one minute before the first run. It fixes three candidate sets:

• the primary set: momentum20, momentum60, momentum120 and cash;
• the primary set plus the equal-weight control;
• momentum rules only.

It also fixes 16 partitions for PBO (8, 10 and 12 as checks), mean label as the main metric (Sharpe as a check), and development data only (2015-01-02 to 2020-12-16, 301 aligned episodes). The reserved 2021 Q1 segment is excluded.

Results (primary set)

Measure	Value
Best candidate	momentum120, Sharpe 0.0254 per five-session episode
Expected best Sharpe from 4 random candidates	0.0177
Chance the true Sharpe is above zero, before adjustment	0.667
DSR	0.552 (well below the usual 0.95 bar)
PBO	0.776 over 12,870 splits
Chance the selected candidate loses out of sample	0.480
Out-of-sample vs in-sample performance slope	-0.727
Walk-forward selected series (249 episodes)	Sharpe -0.0151; chance above zero 0.405

Every check setting gives PBO between 0.76 and 0.96, and DSR stays between 0.55 and 0.58.

Conclusion. The retained candidates look like selection noise, or in-sample rankings that slightly reverse, not skill. This only covers the three-ETF control lane. Nothing was tuned, promoted or rejected.

Limits on what this shows

• The inputs are raw-price labels with a cost deduction, not portfolio returns, so these are not investable Sharpe ratios.
• Only this lane's 4 candidates (3 or 5 in the checks) are counted. Trials run elsewhere in the research programme are not, so this DSR is optimistic.
• momentum60 and momentum120 share about 60% of their labels, so the candidates are far from independent.
• With so few correlated candidates, PBO has few possible values and no p-value.
• The DSR test values come from the paper's own worked example, which I could not re-open offline. No published PBO example with exact answers was available, so the PBO tests use made-up data I constructed.
• No external library's implementation was run for comparison.

Files added: in blueprints/us-equities/overfitting-controls/: overfitting.py, evaluate.py, params.json, crosscheck.py, receipt.json (input hashes, method, parameters, commands, all results) and README.md. The test file is tests/test_overfitting_controls.py. I also added a short section linking to the new lane in blueprints/us-equities/research-evaluation/README.md and updated only that file's entry in manifests/evidence.json. The new files are not registered in the manifest, following the rule to update only changed registered files.

Tests. - New test file (13 tests, 0 skipped), passing on Python 3.13.15:

• the DSR paper's worked example (N=100, V=0.5 annualised, Sharpe 2.5, T=1250, skew -3, kurtosis 10) gives 0.1132 and 0.9004 to four decimals;
• the expected maximum of 100 random normals is within 1.5% of 2.50759;
• PBO checks on made-up data: a full reversal gives 1, a dominant candidate gives 0, pure noise averages about 0.5 over 20 seeds, the fast method matches a brute-force recount, and ties and invalid inputs are handled;
• a replay test recomputes the receipt's results from the retained ledger.
• On the pinned Python 3.12.3 runtime, the new tests plus the existing research-evaluation tests: 25 passed, 0 skipped.
• Full suite (python3 -m unittest on system Python 3.13.15, which lacks numpy, jsonschema and PyYAML): 4031 tests OK, 490 skipped. None of the skips are in the new test file.

Validation. - Required checks on the final tree, all exit 0: scripts/validate.py, validate_catalogs.py, landscape.py --root ., evidence_manifest.py --check, verdict_review_gate.py --base origin/main --root . (no verdict rows changed) and trading_gates.py --check.

• Reproducible output: the evaluation ran three times (twice on Python 3.13.15, once on the pinned 3.12.3 runtime), each exiting 0 with byte-identical output (sha256 3815782f…).
• Independent recomputation: crosscheck.py redoes the calculation with numpy 2.5.3, using direct slicing of all 12,870 splits and its own ranking. It matches the module exactly (DSR 0.5520685371554236, PBO 0.7756021756021756). It shares only the standard library's inverse normal function with the module.
• Input checks: the ledger hash is checked against the research-evaluation receipt, and duplicate records must agree.
• Evidence class: everything is labelled as a local integration check. Nothing here is upstream test evidence.

Independent review. Severities: medium, low, low, low, info, info, info. Fix round 30103cd6: fixed 1, rejected 1 (Partially rejected, the wording that the pooled slope 'can only be negative'. It is pulled towards -1 by construction, but the between-trial part (selected tria).

🤖 Generated with Claude Code