From 12396d369907f67d8d29eea594a4c05319058d86 Mon Sep 17 00:00:00 2001 From: Scout Date: Thu, 24 Sep 2026 12:03:43 -0400 Subject: [PATCH 1/2] Add backtest-overfitting controls (DSR, CSCV PBO) over the research-evaluation ledger Audit gap #11. A standard-library module written from Bailey and Lopez de Prado (2012, 2014) and Bailey, Borwein, Lopez de Prado and Zhu (2017) computes PSR, the deflated Sharpe ratio and the CSCV probability of backtest overfitting. Nothing was vendored. Parameters were frozen in params.json before the first computation. Over the retained 6605-record candidate ledger (sha256 08ea5309..., 301 aligned development episodes), the four selection candidates give DSR 0.552 (momentum120) and PBO 0.776 at S=16. All sensitivity cells give PBO between 0.76 and 0.96. The walk-forward selected series has PSR(0) 0.405. These results show no skill evidence. The work is offline, refetches no data and leaves the reserved segment unscored. Evidence: a local integration check, byte-identical on Python 3.12 and 3.13, with an independent numpy recomputation that matches exactly. The tests cover the DSR paper's numerical example (SR0 0.1132, DSR 0.9004, constants transcribed from the paper), synthetic PBO fixtures and a receipt replay. Co-Authored-By: Claude Opus 5.5 (1M context) --- .../overfitting-controls/README.md | 87 ++ .../overfitting-controls/crosscheck.py | 43 + .../overfitting-controls/evaluate.py | 165 +++ .../overfitting-controls/overfitting.py | 194 +++ .../overfitting-controls/params.json | 70 + .../overfitting-controls/receipt.json | 1138 +++++++++++++++++ .../us-equities/research-evaluation/README.md | 9 + manifests/evidence.json | 4 +- tests/test_overfitting_controls.py | 183 +++ 9 files changed, 1891 insertions(+), 2 deletions(-) create mode 100644 blueprints/us-equities/overfitting-controls/README.md create mode 100644 blueprints/us-equities/overfitting-controls/crosscheck.py create mode 100644 blueprints/us-equities/overfitting-controls/evaluate.py create mode 100644 blueprints/us-equities/overfitting-controls/overfitting.py create mode 100644 blueprints/us-equities/overfitting-controls/params.json create mode 100644 blueprints/us-equities/overfitting-controls/receipt.json create mode 100644 tests/test_overfitting_controls.py diff --git a/blueprints/us-equities/overfitting-controls/README.md b/blueprints/us-equities/overfitting-controls/README.md new file mode 100644 index 000000000..839d7e269 --- /dev/null +++ b/blueprints/us-equities/overfitting-controls/README.md @@ -0,0 +1,87 @@ +# Backtest-overfitting controls + +This lane adds two selection-bias diagnostics to the multi-candidate research +lane and applies them to the retained +[research-evaluation](../research-evaluation/README.md) candidate ledger: + +- **Deflated Sharpe ratio (DSR)**, Bailey and Lopez de Prado (2014): the + probabilistic Sharpe ratio of the best trial, measured against the expected + maximum Sharpe of N null trials rather than zero. +- **Probability of backtest overfitting (PBO)** by combinatorially symmetric + cross-validation (CSCV), Bailey, Borwein, Lopez de Prado and Zhu (2017): the + share of in-sample/out-of-sample block splits in which the in-sample best trial + ranks at or below the out-of-sample median. + +[`overfitting.py`](overfitting.py) implements both from the papers using only the +Python standard library. Nothing was copied or vendored. The citations and the +formulas are in its docstring. [`evaluate.py`](evaluate.py) applies the module +offline to the hash-pinned ledger using the parameters in +[`params.json`](params.json). Those parameters were frozen before the first +computation. [`crosscheck.py`](crosscheck.py) recomputes the primary DSR and PBO +independently with numpy. + +## Result — September 24, 2026 + +Input: the 6,605-record candidate ledger (SHA-256 `08ea5309…2f42672`, equal to +the research-evaluation receipt's `ledger_sha256`). The in-tree copy is the +byte-identical gap-wave2 regeneration. No data was refetched. After +deduplication, the development segment has 301 aligned five-session episodes per +candidate, from 2015-01-02 to 2020-12-16. The reserved 2021Q1 segment is +excluded. + +| Diagnostic (frozen four-candidate selection set) | Value | +| --- | ---: | +| Best per-episode Sharpe (momentum120), not annualized | 0.0254 | +| Expected maximum null Sharpe SR0 (N = 4) | 0.0177 | +| PSR against zero | 0.667 | +| **Deflated Sharpe ratio** | **0.552** | +| **PBO** (S = 16, mean-label metric, 12,870 splits) | **0.776** | +| Probability that the selected trial loses out of sample | 0.480 | +| Out-of-sample on in-sample degradation slope | -0.727 | +| Walk-forward selected series: Sharpe / PSR(0), 249 episodes | -0.0151 / 0.405 | + +All sensitivity cells give PBO (lambda <= 0) between 0.76 and 0.96. The cells +cover S = 8, 10, 12 and 16, the mean and Sharpe metrics, and three trial sets: +the primary four, adding the equal-weight control, and momentum rules only. +DSR stays between 0.55 and 0.58. The [receipt](receipt.json) retains every cell, +the strict lambda < 0 shares, input and source hashes, commands and the +independent recomputation. + +**Conclusion.** These retained candidates show no deflated-Sharpe evidence of +skill. The in-sample best candidate usually ranks below the out-of-sample +median. The evidence is consistent with selection noise, or mild +anti-persistence of in-sample ranking, not skill. This negative diagnostic +covers only this three-ETF control lane. It does not tune, promote or reject any +future strategy. + +## Boundaries + +- The inputs are raw-price five-session labels less a fixed 20bp proxy, not + portfolio returns. The Sharpe values are per-episode label ratios. +- N counts only this lane's frozen candidates. The research program ran other + trials, so relative to all of them this DSR is optimistic. No effective-N + clustering was applied. momentum60 and momentum120 share about 60% of their + development labels. +- With three to five correlated trials, CSCV has only a few distinct ranks. + Contiguous 18-episode blocks are assumed exchangeable. PBO is descriptive and + has no p-value. +- Evidence class: a local integration check with an independent numpy + recomputation. The DSR unit test reproduces the paper's numerical example + (SR0 0.1132, DSR 0.9004). Those constants are transcribed from the paper, + which was not re-fetched offline. The PBO tests are locally constructed + synthetic fixtures. No upstream implementation was executed. + +## Reproduce + +```sh +python3 blueprints/us-equities/overfitting-controls/evaluate.py +python3 -m unittest tests.test_overfitting_controls -v +# optional independent recomputation (needs numpy) +python blueprints/us-equities/overfitting-controls/crosscheck.py +``` + +Reusing the module for another candidate ledger: pass a T x N matrix of +per-period trial performance to `cscv_pbo`. Pass the selected trial's Sharpe, +moments, trial count and cross-trial Sharpe variance to `deflated_sharpe`. +Freeze the trial set and the parameters before looking at the results. Count +every trial that was actually tried. diff --git a/blueprints/us-equities/overfitting-controls/crosscheck.py b/blueprints/us-equities/overfitting-controls/crosscheck.py new file mode 100644 index 000000000..a95637afc --- /dev/null +++ b/blueprints/us-equities/overfitting-controls/crosscheck.py @@ -0,0 +1,43 @@ +"""Independent recomputation with numpy (different code path from overfitting.py); normal quantiles from statistics.NormalDist, CDF from math.erf.""" +import json, sys, itertools, math +from decimal import Decimal +import numpy as np +from statistics import NormalDist +ppf = NormalDist().inv_cdf +cdf = lambda z: 0.5 * (1 + math.erf(z / math.sqrt(2))) + +def rankdata(a): + a = np.asarray(a); r = np.empty(len(a)) + for i, x in enumerate(a): + r[i] = (a < x).sum() + ((a == x).sum() + 1) / 2 + return r + +root = sys.argv[1] if len(sys.argv) > 1 else "." +ledger = json.load(open(root + "/evidence/artifacts/gap-wave2-20260923/us-equities__portfolio-risk/raw/3/regenerated/run-1/candidate-ledger.json")) +trials = ["momentum20", "momentum60", "momentum120", "cash"] +v = {} +for r in ledger: + if r["status"] == "observed" and r["decision"] <= "2020-12-31": + v[(r["candidate"], r["decision"])] = float(Decimal(r["net_proxy"])) +dates = sorted({d for c, d in v}) +M = np.array([[v[(c, d)] for c in trials] for d in dates]) +sd = M.std(axis=0) +sr = np.where(sd > 0, M.mean(axis=0) / np.where(sd > 0, sd, 1), 0.0) +b = int(np.argmax(sr)); x = M[:, b]; T = len(x) +d = x - x.mean(); g3 = (d**3).mean() / (d**2).mean()**1.5; g4 = (d**4).mean() / (d**2).mean()**2 +V = np.var(sr, ddof=1); N = len(trials); g = np.euler_gamma +sr0 = math.sqrt(V) * ((1 - g) * ppf(1 - 1 / N) + g * ppf(1 - 1 / (N * np.e))) +dsr = cdf((sr[b] - sr0) * math.sqrt(T - 1) / math.sqrt(1 - g3 * sr[b] + (g4 - 1) / 4 * sr[b] ** 2)) +# PBO, S=16, mean metric, direct slicing +S = 16; size = T // S; U = M[T - size * S:]; blocks = np.split(U, S) +lam = [] +for J in itertools.combinations(range(S), S // 2): + tr = np.vstack([blocks[i] for i in J]); te = np.vstack([blocks[i] for i in range(S) if i not in J]) + ism, osm = tr.mean(0), te.mean(0) + n = int(np.flatnonzero(ism == ism.max())[0]) + w = rankdata(osm)[n] / (len(trials) + 1) + lam.append(math.log(w / (1 - w))) +lam = np.array(lam) +print(json.dumps({"T": T, "selected": trials[b], "sr": sr[b], "sr0": sr0, "dsr": dsr, + "pbo": float((lam <= 0).mean()), "combinations": len(lam), + "numpy": np.__version__}, indent=1)) diff --git a/blueprints/us-equities/overfitting-controls/evaluate.py b/blueprints/us-equities/overfitting-controls/evaluate.py new file mode 100644 index 000000000..b1c782fa6 --- /dev/null +++ b/blueprints/us-equities/overfitting-controls/evaluate.py @@ -0,0 +1,165 @@ +#!/usr/bin/env python3 +"""Deflated Sharpe ratio and CSCV PBO over the retained research-evaluation ledger. + +Offline: reads only the hash-pinned ledger, the research-evaluation receipt and +params.json. Prints deterministic results JSON (no timestamps) to stdout. +""" +import argparse +from decimal import Decimal +import hashlib +import importlib.util +import json +import math +from pathlib import Path + +HERE = Path(__file__).resolve().parent +ROOT = HERE.parents[2] + + +def _library(): + spec = importlib.util.spec_from_file_location("overfitting_controls", HERE / "overfitting.py") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +of = _library() + + +def sha256(path): + return hashlib.sha256(Path(path).read_bytes()).hexdigest() + + +def pointer(document, path): + node = document + for token in path.strip("/").split("/"): + node = node[int(token)] if isinstance(node, list) else node[token] + return node + + +def load_inputs(root, params): + src = params["source"] + ledger_path, receipt_path = root / src["ledger_path"], root / src["research_receipt_path"] + if sha256(ledger_path) != src["ledger_sha256"]: + raise ValueError("ledger hash mismatch") + receipt = json.loads(receipt_path.read_text()) + if pointer(receipt, src["research_receipt_ledger_pointer"]) != src["ledger_sha256"]: + raise ValueError("research receipt does not name this ledger") + ledger = json.loads(ledger_path.read_text()) + if len(ledger) != src["ledger_records"]: + raise ValueError("ledger record count mismatch") + return ledger, receipt + + +def episode_values(ledger, development_end): + """One net_proxy per (candidate, decision) in the development segment; conflicts are errors.""" + values = {} + for row in ledger: + if row["status"] != "observed" or row["decision"] > development_end: + continue + key = (row["candidate"], row["decision"]) + value = Decimal(row["net_proxy"]) + if key in values and values[key] != value: + raise ValueError("conflicting duplicate episode label: %s %s" % key) + values[key] = value + return values + + +def aligned_matrix(values, trials): + dates = sorted({d for (c, d) in values if c in trials}) + missing = [(c, d) for c in trials for d in dates if (c, d) not in values] + if missing or not dates: + raise ValueError("trial series are not aligned") + return dates, [[float(values[(c, d)]) for c in trials] for d in dates] + + +def dsr_block(trials, matrix): + columns = list(zip(*matrix)) + per_trial = {} + for name, column in zip(trials, columns): + m = of.moments(column) + per_trial[name] = dict(m, sharpe=of.sharpe_from(m["mean"], m["std"])) + srs = [per_trial[t]["sharpe"] for t in trials] + best = max(range(len(trials)), key=lambda i: (srs[i], -i)) + selected = per_trial[trials[best]] + mean_sr = math.fsum(srs) / len(srs) + variance = math.fsum((s - mean_sr) ** 2 for s in srs) / (len(srs) - 1) + dsr, sr0 = of.deflated_sharpe(selected["sharpe"], selected["n"], selected["skew"], + selected["kurtosis"], len(trials), variance) + psr0 = of.probabilistic_sharpe(selected["sharpe"], selected["n"], selected["skew"], selected["kurtosis"], 0.0) + return {"trials": list(trials), "observations": len(matrix), "per_trial": per_trial, + "selected_trial": trials[best], "selected_sharpe": selected["sharpe"], + "trial_sharpe_variance_ddof1": variance, "expected_max_sharpe_sr0": sr0, + "psr_vs_zero": psr0, "dsr": dsr} + + +def pbo_block(matrix, partitions, metric): + result = of.cscv_pbo(matrix, partitions, metric) + logits = result.pop("logits") + histogram = {} + for x in logits: + key = "%.6f" % x + histogram[key] = histogram.get(key, 0) + 1 + result["lambda_histogram"] = dict(sorted(histogram.items(), key=lambda kv: float(kv[0]))) + return result + + +def walk_forward_block(ledger, receipt): + folds = receipt["results"]["development_folds"] + chosen = {f["id"]: f["chosen"] for f in folds} + series = sorted((r["decision"], float(Decimal(r["net_proxy"])), r["fold"]) + for r in ledger if r["phase"] == "evaluation" and r["status"] == "observed" + and r["fold"] in chosen and r["candidate"] == chosen[r["fold"]]) + decisions = [d for d, _, _ in series] + if len(set(decisions)) != len(decisions): + raise ValueError("overlapping walk-forward evaluation episodes") + values = [v for _, v, _ in series] + m = of.moments(values) + sr = of.sharpe_from(m["mean"], m["std"]) + return {"folds": len(chosen), "chosen_counts": {c: list(chosen.values()).count(c) for c in sorted(set(chosen.values()))}, + "observations": len(values), "first_decision": decisions[0], "last_decision": decisions[-1], + "moments": m, "sharpe": sr, "psr_vs_zero": of.probabilistic_sharpe(sr, m["n"], m["skew"], m["kurtosis"], 0.0)} + + +def compute(root=ROOT, params_path=HERE / "params.json"): + params = json.loads(Path(params_path).read_text()) + ledger, receipt = load_inputs(root, params) + values = episode_values(ledger, params["series"]["development_end"]) + dsr, pbo = {}, {} + grid = [params["pbo"]["partitions_primary"]] + params["pbo"]["partitions_sensitivity"] + metrics = [params["pbo"]["metric_primary"]] + params["pbo"]["metric_sensitivity"] + dates = None + for name, trials in params["trial_sets"].items(): + set_dates, matrix = aligned_matrix(values, trials) + if dates is None: + dates = set_dates + elif dates != set_dates: + raise ValueError("trial sets cover different decision dates") + dsr[name] = dsr_block(trials, matrix) + pbo[name] = {"%s_S%d" % (metric, s): pbo_block(matrix, s, metric) for metric in metrics for s in grid} + primary = pbo["primary"]["%s_S%d" % (params["pbo"]["metric_primary"], params["pbo"]["partitions_primary"])] + return { + "inputs": {"params_sha256": sha256(params_path), + "ledger_sha256": sha256(root / params["source"]["ledger_path"]), + "research_receipt_sha256": sha256(root / params["source"]["research_receipt_path"]), + "overfitting_py_sha256": sha256(HERE / "overfitting.py"), + "evaluate_py_sha256": sha256(HERE / "evaluate.py")}, + "ledger_records": len(ledger), + "development_episodes": len(dates), "first_decision": dates[0], "last_decision": dates[-1], + "dsr": dsr, "pbo": pbo, + "primary": {"dsr": dsr["primary"]["dsr"], "dsr_selected_trial": dsr["primary"]["selected_trial"], + "pbo": primary["pbo"], "pbo_strict": primary["pbo_strict"], + "probability_oos_loss": primary["probability_oos_loss"]}, + "walk_forward": walk_forward_block(ledger, receipt), + } + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--root", type=Path, default=ROOT) + args = parser.parse_args() + print(json.dumps(compute(args.root.resolve()), indent=2, sort_keys=True)) + + +if __name__ == "__main__": + main() diff --git a/blueprints/us-equities/overfitting-controls/overfitting.py b/blueprints/us-equities/overfitting-controls/overfitting.py new file mode 100644 index 000000000..bea15b47a --- /dev/null +++ b/blueprints/us-equities/overfitting-controls/overfitting.py @@ -0,0 +1,194 @@ +"""Backtest-overfitting controls written from the primary papers (standard library only). + +Sources (formulas transcribed from the papers; no third-party code was copied): + +* Probabilistic Sharpe ratio (PSR): D. H. Bailey and M. Lopez de Prado, + "The Sharpe Ratio Efficient Frontier", Journal of Risk 15(2), 2012, 3-44. + PSR(SR*) = Phi((SR - SR*) * sqrt(T - 1) / sqrt(1 - g3 * SR + (g4 - 1) / 4 * SR^2)), + with g3 the skewness and g4 the (non-excess) kurtosis of the returns. +* Deflated Sharpe ratio (DSR): D. H. Bailey and M. Lopez de Prado, "The Deflated + Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and + Non-Normality", Journal of Portfolio Management 40(5), 2014, 94-107 + (SSRN 2460551). DSR = PSR(SR0) with + SR0 = sqrt(V[SR_n]) * ((1 - gamma) * Phi^-1(1 - 1/N) + gamma * Phi^-1(1 - 1/(N e))), + gamma the Euler-Mascheroni constant, N the number of independent trials. +* Probability of backtest overfitting (PBO) by combinatorially symmetric + cross-validation (CSCV): D. H. Bailey, J. M. Borwein, M. Lopez de Prado and + Q. J. Zhu, "The Probability of Backtest Overfitting", Journal of + Computational Finance 20(4), 2017, 39-69 (SSRN 2326253). + +Sharpe ratios here are per-observation (not annualized) unless a caller scales them. +""" +from __future__ import annotations + +from itertools import combinations +import math +from statistics import NormalDist + +EULER_GAMMA = 0.5772156649015329 +_NORMAL = NormalDist() + + +def moments(values): + """Population mean, standard deviation, skewness and non-excess kurtosis.""" + xs = [float(x) for x in values] + n = len(xs) + if n < 2 or not all(math.isfinite(x) for x in xs): + raise ValueError("need at least two finite observations") + mean = math.fsum(xs) / n + dev = [x - mean for x in xs] + m2 = math.fsum(d * d for d in dev) / n + if m2 == 0.0: + return {"n": n, "mean": mean, "std": 0.0, "skew": None, "kurtosis": None} + m3 = math.fsum(d ** 3 for d in dev) / n + m4 = math.fsum(d ** 4 for d in dev) / n + return {"n": n, "mean": mean, "std": math.sqrt(m2), "skew": m3 / m2 ** 1.5, "kurtosis": m4 / m2 ** 2} + + +def sharpe_from(mean, std): + if std > 0.0: + return mean / std + if mean == 0.0: + return 0.0 + raise ValueError("zero-variance series with nonzero mean has no finite Sharpe ratio") + + +def sharpe(values): + stats = moments(values) + return sharpe_from(stats["mean"], stats["std"]) + + +def probabilistic_sharpe(sr, t, skew, kurtosis, sr_benchmark=0.0): + """PSR: probability that the true Sharpe exceeds sr_benchmark (Bailey and Lopez de Prado 2012).""" + if t < 2: + raise ValueError("need at least two observations") + variance_term = 1.0 - skew * sr + (kurtosis - 1.0) / 4.0 * sr * sr + if not variance_term > 0.0: + raise ValueError("non-positive Sharpe variance term") + z = (sr - sr_benchmark) * math.sqrt(t - 1) / math.sqrt(variance_term) + return _NORMAL.cdf(z) + + +def expected_max_sharpe(trials, variance): + """Expected maximum Sharpe of `trials` independent null trials (Bailey and Lopez de Prado 2014).""" + if not isinstance(trials, int) or trials < 1: + raise ValueError("trials must be a positive integer") + if not variance >= 0.0: + raise ValueError("variance must be nonnegative") + if trials == 1: + return 0.0 # no selection among trials: the null benchmark is zero + return math.sqrt(variance) * ((1.0 - EULER_GAMMA) * _NORMAL.inv_cdf(1.0 - 1.0 / trials) + + EULER_GAMMA * _NORMAL.inv_cdf(1.0 - 1.0 / (trials * math.e))) + + +def deflated_sharpe(sr, t, skew, kurtosis, trials, trial_variance): + """DSR = PSR evaluated at the expected maximum null Sharpe; returns (dsr, sr0).""" + sr0 = expected_max_sharpe(trials, trial_variance) + return probabilistic_sharpe(sr, t, skew, kurtosis, sr0), sr0 + + +def _merge(a, b): + """Chan et al. parallel combination of (count, mean, M2) aggregates.""" + n = a[0] + b[0] + delta = b[1] - a[1] + mean = a[1] + delta * b[0] / n + return (n, mean, a[2] + b[2] + delta * delta * a[0] * b[0] / n) + + +def _aggregate(rows): + n = len(rows) + mean = math.fsum(rows) / n + return (n, mean, math.fsum((x - mean) ** 2 for x in rows)) + + +def _score(aggregate, metric): + n, mean, m2 = aggregate + if metric == "mean": + return mean + if metric == "sharpe": + std = math.sqrt(max(m2, 0.0) / n) + if std <= 1e-12 * max(1.0, abs(mean)): + std = 0.0 # merged M2 of a constant column can carry rounding residue + return sharpe_from(mean, std) + raise ValueError("metric must be 'mean' or 'sharpe'") + + +def _average_ranks(values): + order = sorted(range(len(values)), key=lambda i: values[i]) + ranks = [0.0] * len(values) + i = 0 + while i < len(order): + j = i + while j + 1 < len(order) and values[order[j + 1]] == values[order[i]]: + j += 1 + for k in range(i, j + 1): + ranks[order[k]] = (i + j) / 2.0 + 1.0 + i = j + 1 + return ranks + + +def cscv_pbo(matrix, partitions=16, metric="mean"): + """Probability of backtest overfitting by CSCV. + + matrix: T rows (chronological observations) by N columns (trials). + The earliest T mod S rows are dropped so the S contiguous blocks are equal. + For every choice of S/2 blocks as the in-sample set J (complement J-bar out of + sample): n* = argmax in-sample metric (ties by column order), omega = average + ascending out-of-sample rank of n* / (N + 1), lambda = logit(omega). + PBO is the share of combinations with lambda <= 0. + """ + rows = [[float(x) for x in row] for row in matrix] + if not rows or any(len(r) != len(rows[0]) for r in rows): + raise ValueError("matrix must be rectangular and nonempty") + t, n = len(rows), len(rows[0]) + if n < 2: + raise ValueError("need at least two trials") + if not isinstance(partitions, int) or partitions < 2 or partitions % 2: + raise ValueError("partitions must be an even integer >= 2") + size = t // partitions + if size < 2: + raise ValueError("each partition needs at least two rows") + if not all(math.isfinite(x) for r in rows for x in r): + raise ValueError("nonfinite performance value") + dropped = t - size * partitions + used = rows[dropped:] + blocks = [[_aggregate([used[b * size + i][c] for i in range(size)]) for c in range(n)] + for b in range(partitions)] + + def combine(indices, column): + agg = blocks[indices[0]][column] + for b in indices[1:]: + agg = _merge(agg, blocks[b][column]) + return agg + + logits, selected_is, selected_oos, chosen_counts = [], [], [], [0] * n + for chosen_blocks in combinations(range(partitions), partitions // 2): + rest = tuple(b for b in range(partitions) if b not in chosen_blocks) + in_sample = [_score(combine(chosen_blocks, c), metric) for c in range(n)] + out_sample = [_score(combine(rest, c), metric) for c in range(n)] + best = max(range(n), key=lambda c: (in_sample[c], -c)) + omega = _average_ranks(out_sample)[best] / (n + 1) + logits.append(math.log(omega / (1.0 - omega))) + selected_is.append(in_sample[best]) + selected_oos.append(out_sample[best]) + chosen_counts[best] += 1 + count = len(logits) + at_or_below = sum(x <= 0.0 for x in logits) + zero = sum(x == 0.0 for x in logits) + mean_is = math.fsum(selected_is) / count + mean_oos = math.fsum(selected_oos) / count + sxx = math.fsum((x - mean_is) ** 2 for x in selected_is) + slope = (math.fsum((x - mean_is) * (y - mean_oos) for x, y in zip(selected_is, selected_oos)) / sxx + if sxx > 0 else None) + return { + "partitions": partitions, "metric": metric, "rows": t, "rows_used": len(used), + "rows_dropped_earliest": dropped, "rows_per_partition": size, "trials": n, + "combinations": count, "pbo": at_or_below / count, "pbo_strict": (at_or_below - zero) / count, + "lambda_zero_count": zero, "lambda_mean": math.fsum(logits) / count, + "lambda_min": min(logits), "lambda_max": max(logits), + "probability_oos_loss": sum(y < 0.0 for y in selected_oos) / count, + "selected_in_sample_mean": mean_is, "selected_out_of_sample_mean": mean_oos, + "degradation_slope": slope, + "degradation_intercept": (mean_oos - slope * mean_is) if slope is not None else None, + "in_sample_selection_counts": chosen_counts, "logits": logits, + } diff --git a/blueprints/us-equities/overfitting-controls/params.json b/blueprints/us-equities/overfitting-controls/params.json new file mode 100644 index 000000000..7618d4c6b --- /dev/null +++ b/blueprints/us-equities/overfitting-controls/params.json @@ -0,0 +1,70 @@ +{ + "schema_version": 1, + "id": "overfitting-controls-research-evaluation-v1", + "frozen_before_computation": true, + "purpose": "Deflated Sharpe ratio and CSCV probability of backtest overfitting over the retained multi-candidate research-evaluation ledger. Diagnostic only; no selection, tuning or promotion follows from the result.", + "source": { + "ledger_path": "evidence/artifacts/gap-wave2-20260923/us-equities__portfolio-risk/raw/3/regenerated/run-1/candidate-ledger.json", + "ledger_sha256": "08ea5309466045574ba4b9c8524608ed66c91edb22cb592a5bad84e1e2f42672", + "ledger_records": 6605, + "research_receipt_path": "blueprints/us-equities/research-evaluation/receipt.json", + "research_receipt_ledger_pointer": "/results/ledger_sha256", + "provenance": "Byte-identical regeneration of the private run-1 ledger whose SHA-256 the research-evaluation receipt records; the regeneration comparison is the gap-wave2 portfolio-risk raw/3 compare.json." + }, + "series": { + "development_end": "2020-12-31", + "segment": "development only: observed labels whose decision date is on or before 2020-12-31; reserved 2021Q1 labels are excluded", + "value": "net_proxy (five-session open-to-open raw-price label less the fixed 20bp cost proxy), parsed from its decimal string", + "deduplication": "One value per (candidate, decision). Training and evaluation copies of the same episode must be identical; any conflict is an error.", + "alignment": "Every trial must have an observed value at every retained decision date; otherwise an error." + }, + "trial_sets": { + "primary": [ + "momentum20", + "momentum60", + "momentum120", + "cash" + ], + "with_equalweight_control": [ + "momentum20", + "momentum60", + "momentum120", + "cash", + "equalweight" + ], + "momentum_only": [ + "momentum20", + "momentum60", + "momentum120" + ] + }, + "dsr": { + "sharpe": "Per-episode (five-session) mean divided by population standard deviation; not annualized. A zero-variance series with zero mean (cash) has Sharpe 0.", + "moments": "Population skewness and non-excess kurtosis of the selected trial's series (Bailey and Lopez de Prado 2014 gamma3, gamma4).", + "selected_trial": "Highest full-development Sharpe within the trial set; ties by trial-set order.", + "trial_count": "Number of trials in the set (no effective-N clustering).", + "trial_variance": "Sample variance (ddof=1) of the trial Sharpe estimates.", + "benchmark": "Expected maximum Sharpe under the null, SR0 = sqrt(V) * ((1-gamma) * Phi^-1(1-1/N) + gamma * Phi^-1(1-1/(N e)))." + }, + "pbo": { + "method": "CSCV (Bailey, Borwein, Lopez de Prado and Zhu 2017)", + "partitions_primary": 16, + "partitions_sensitivity": [ + 8, + 10, + 12 + ], + "metric_primary": "mean", + "metric_sensitivity": [ + "sharpe" + ], + "truncation": "Drop the earliest T mod S rows so that S contiguous blocks have equal size.", + "in_sample_tie_break": "Trial-set order (the frozen plan's tie rule).", + "out_of_sample_rank": "Ascending rank among the N trials with average ranks for ties; omega = rank / (N + 1); lambda = ln(omega / (1 - omega)).", + "pbo_rule": "Share of combinations with lambda <= 0 (primary). The strict lambda < 0 share and the count of lambda == 0 are reported alongside." + }, + "walk_forward": { + "series": "Development evaluation labels of each development fold's frozen chosen candidate (from the research-evaluation receipt), observed only.", + "statistic": "Probabilistic Sharpe ratio against a zero benchmark; the walk-forward selector is one pre-registered procedure, so no additional trial deflation is applied to it." + } +} diff --git a/blueprints/us-equities/overfitting-controls/receipt.json b/blueprints/us-equities/overfitting-controls/receipt.json new file mode 100644 index 000000000..07aa87680 --- /dev/null +++ b/blueprints/us-equities/overfitting-controls/receipt.json @@ -0,0 +1,1138 @@ +{ + "schema_version": 1, + "id": "overfitting-controls-research-evaluation-20260924", + "evidence_class": "local_integration_check", + "observed_utc": "2026-09-24T15:54:19Z", + "status": "diagnostic_computed_no_promotion", + "claim": "Deflated Sharpe ratio (Bailey and Lopez de Prado 2014) and CSCV probability of backtest overfitting (Bailey, Borwein, Lopez de Prado and Zhu 2017) were computed offline over the 6605-record research-evaluation candidate ledger (301 aligned development episodes, 2015-01-02 to 2020-12-16). For the frozen four-candidate selection set the highest-Sharpe candidate (momentum120, per-episode Sharpe 0.0254) has DSR 0.5521 against SR0 0.0177, and PBO is 0.7756 (S=16, mean-label metric, 12870 combinations). The walk-forward selected series has Sharpe -0.0151 and PSR(0) 0.4048. The retained candidates show no deflated-Sharpe evidence of skill, and the in-sample best candidate usually ranks below the out-of-sample median.", + "frozen_parameters": { + "path": "blueprints/us-equities/overfitting-controls/params.json", + "sha256": "0e371da46601c673453a40ebcb608e38512348c37bf95a63ac7793d263c21487", + "frozen_utc": "2026-09-24T15:53:14Z", + "before": "The first computation over the retained ledger (run-1 started 2026-09-24T15:54:19Z). Library unit tests on synthetic fixtures and the published example ran before it; no parameter changed after results were seen." + }, + "inputs": { + "ledger": { + "path": "evidence/artifacts/gap-wave2-20260923/us-equities__portfolio-risk/raw/3/regenerated/run-1/candidate-ledger.json", + "sha256": "08ea5309466045574ba4b9c8524608ed66c91edb22cb592a5bad84e1e2f42672", + "bytes": 2536508, + "records": 6605, + "provenance": "SHA-256 equals results.ledger_sha256 in blueprints/us-equities/research-evaluation/receipt.json (native skfolio 1.2.9 run of 2026-09-19); this in-tree copy is the gap-wave2 portfolio-risk byte-identical regeneration (raw/3/compare.json). No data was refetched." + }, + "research_receipt": { + "path": "blueprints/us-equities/research-evaluation/receipt.json", + "sha256": "92a2c4c1984c7cd38af21fbc9b5290395805fed3177522b19d408ad50c59d56b", + "used_for": "Ledger hash binding and each development fold's frozen chosen candidate for the walk-forward series." + } + }, + "method": { + "primary_sources": [ + { + "citation": "Bailey, D. H. and Lopez de Prado, M. (2012). The Sharpe Ratio Efficient Frontier. Journal of Risk 15(2), 3-44.", + "used_for": "Probabilistic Sharpe ratio with skewness and kurtosis adjustment." + }, + { + "citation": "Bailey, D. H. and Lopez de Prado, M. (2014). The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-Normality. Journal of Portfolio Management 40(5), 94-107. SSRN 2460551.", + "used_for": "Expected maximum null Sharpe SR0 and DSR = PSR(SR0); the paper's numerical example is a unit test." + }, + { + "citation": "Bailey, D. H., Borwein, J. M., Lopez de Prado, M. and Zhu, Q. J. (2017). The Probability of Backtest Overfitting. Journal of Computational Finance 20(4), 39-69. SSRN 2326253.", + "used_for": "CSCV algorithm, relative rank logits, PBO, probability of out-of-sample loss and performance degradation regression." + } + ], + "implementation_decision": { + "choice": "Written from the papers as a standard-library module (overfitting.py); no third-party code copied or vendored.", + "alternatives_considered": [ + "eslazarev/purged-cross-validation: catalogued discovery_only in catalogs/landscape/claude-independent-discovery.json with no source pin, licence review or execution in this repository.", + "pypbo (Python) and the R pbo package: no copy on this host and no reviewed pin; installing either would require a network fetch and a new dependency lock for roughly 150 lines of arithmetic.", + "skfolio 1.2.9 (already locked for this lane) supplied the WalkForward splitter upstream of this ledger; this change did not review skfolio for DSR or PBO estimators." + ], + "overturn_condition": "Prefer a maintained, licence-reviewed upstream implementation if, pinned and executed, it reproduces the DSR numerical example and these ledger results within floating-point tolerance; also revisit if the papers' text contradicts a transcribed formula." + }, + "parameters": "See frozen params.json: trial sets, per-episode (non-annualized) Sharpe with population moments, ddof=1 trial variance, S=16 primary with 8/10/12 sensitivity, mean-label primary metric with Sharpe sensitivity, earliest-row truncation, trial-order in-sample ties, average out-of-sample ranks, PBO rule lambda <= 0." + }, + "implementation": { + "overfitting.py": { + "path": "blueprints/us-equities/overfitting-controls/overfitting.py", + "sha256": "c44a5f7d00f8cb2b9787ab8f408e76731a99e4ac7e67f69f3efbb34b4b7eebdf" + }, + "evaluate.py": { + "path": "blueprints/us-equities/overfitting-controls/evaluate.py", + "sha256": "e62af085613c0b341ffdfc2a0eac94255551325f430844f29e317a26ede6cedc" + }, + "crosscheck.py": { + "path": "blueprints/us-equities/overfitting-controls/crosscheck.py", + "sha256": "dc03b73d54ecf64f4286e98cab7242898979a05247c2042729efe80b663d03d0" + } + }, + "commands": [ + { + "id": "run-1", + "argv": [ + "python3", + "blueprints/us-equities/overfitting-controls/evaluate.py" + ], + "cwd": "checkout root", + "runtime": "CPython 3.13.15 (standard library only)", + "started_utc": "2026-09-24T15:54:19Z", + "ended_utc": "2026-09-24T15:54:21Z", + "exit": 0, + "stderr": "only the elapsed-time line of the wrapping /usr/bin/time (1.54 s)", + "stdout_sha256": "3815782f85fcb7c8aebd6defce82ae9540039f6bf408a82a83ff47dd0a83ff43" + }, + { + "id": "run-2", + "argv": [ + "python3", + "blueprints/us-equities/overfitting-controls/evaluate.py" + ], + "runtime": "CPython 3.13.15", + "exit": 0, + "stderr_bytes": 0, + "stdout_sha256": "3815782f85fcb7c8aebd6defce82ae9540039f6bf408a82a83ff47dd0a83ff43" + }, + { + "id": "run-3", + "argv": [ + "", + "blueprints/us-equities/overfitting-controls/evaluate.py" + ], + "runtime": "CPython 3.12.3", + "exit": 0, + "stderr_bytes": 0, + "stdout_sha256": "3815782f85fcb7c8aebd6defce82ae9540039f6bf408a82a83ff47dd0a83ff43" + } + ], + "determinism": "All three runs produced byte-identical stdout; the results object below is that stdout.", + "independent_observation": { + "method": "crosscheck.py recomputes the primary DSR and PBO with numpy 2.5.3 (array moments, direct row slicing of all 12870 CSCV combinations, explicit average ranks, math.erf CDF), sharing only statistics.NormalDist.inv_cdf and the ledger with the module under test.", + "runtime": "CPython 3.12.3 pinned adaptive-paper runtime, exit 0, empty stderr", + "stdout_sha256": "16fa24af60088749727bea5f789136a44d03be78e68dd018ce6a0db788a63395", + "result": { + "T": 301, + "selected": "momentum120", + "sr": 0.02539620742431157, + "sr0": 0.017690872785666837, + "dsr": 0.5520685371554236, + "pbo": 0.7756021756021756, + "combinations": 12870, + "numpy": "2.5.3" + }, + "matches_primary": true + }, + "tests": { + "file": "tests/test_overfitting_controls.py", + "command": "python3 -m unittest tests.test_overfitting_controls -v", + "classes": { + "published_worked_example": "Bailey and Lopez de Prado (2014) numerical example (N=100, annualized V=1/2, SR=2.5, T=1250, skew -3, kurtosis 10): SR0 0.1132 and DSR 0.9004 reproduced to four decimals. Expected values are transcribed from the paper; the paper text was not re-fetched in this offline session.", + "reference_value": "Expected maximum of 100 iid standard normals (order-statistic tables, 2.50759) within the approximation's 1.5% tolerance.", + "synthetic_fixture": "Locally constructed PBO cases: exact reversal (PBO 1), dominant trial (PBO 0), pure-noise mean near 0.5 over 20 seeds, block-merge equality with a brute-force recomputation, tie handling and invalid configurations. No published PBO numerical example with exact expected values was available offline.", + "replay": "Recomputes this receipt's results object from the retained ledger (float tolerance 1e-9 relative)." + }, + "observed": { + "focused": "13 tests passed, 0 skipped on CPython 3.13.15; with tests.test_research_evaluation, 25 passed, 0 skipped on CPython 3.12.3 (pinned runtime)", + "full_suite": "python3 -m unittest on CPython 3.13.15 (system interpreter without numpy, jsonschema or PyYAML): 4031 tests OK, 490 skipped; none of the skips are in tests.test_overfitting_controls" + } + }, + "results": { + "development_episodes": 301, + "dsr": { + "momentum_only": { + "dsr": 0.5608136084061657, + "expected_max_sharpe_sr0": 0.016387328850452767, + "observations": 301, + "per_trial": { + "momentum120": { + "kurtosis": 11.70581994534825, + "mean": 0.000647213217448946, + "n": 301, + "sharpe": 0.02539620742431157, + "skew": -1.4941526859828613, + "std": 0.025484640546342932 + }, + "momentum20": { + "kurtosis": 10.534059391680314, + "mean": 0.00047619722098306353, + "n": 301, + "sharpe": 0.020820777225615553, + "skew": -0.6864816724012911, + "std": 0.022871250953936714 + }, + "momentum60": { + "kurtosis": 14.863770264503078, + "mean": -0.00022989605827723083, + "n": 301, + "sharpe": -0.00993755279683209, + "skew": -2.2018993871687806, + "std": 0.02313407163487146 + } + }, + "psr_vs_zero": 0.6669116476097554, + "selected_sharpe": 0.02539620742431157, + "selected_trial": "momentum120", + "trial_sharpe_variance_ddof1": 0.0003692473397730362, + "trials": [ + "momentum20", + "momentum60", + "momentum120" + ] + }, + "primary": { + "dsr": 0.5520685371554236, + "expected_max_sharpe_sr0": 0.01769087278566683, + "observations": 301, + "per_trial": { + "cash": { + "kurtosis": null, + "mean": 0.0, + "n": 301, + "sharpe": 0.0, + "skew": null, + "std": 0.0 + }, + "momentum120": { + "kurtosis": 11.70581994534825, + "mean": 0.000647213217448946, + "n": 301, + "sharpe": 0.02539620742431157, + "skew": -1.4941526859828613, + "std": 0.025484640546342932 + }, + "momentum20": { + "kurtosis": 10.534059391680314, + "mean": 0.00047619722098306353, + "n": 301, + "sharpe": 0.020820777225615553, + "skew": -0.6864816724012911, + "std": 0.022871250953936714 + }, + "momentum60": { + "kurtosis": 14.863770264503078, + "mean": -0.00022989605827723083, + "n": 301, + "sharpe": -0.00993755279683209, + "skew": -2.2018993871687806, + "std": 0.02313407163487146 + } + }, + "psr_vs_zero": 0.6669116476097554, + "selected_sharpe": 0.02539620742431157, + "selected_trial": "momentum120", + "trial_sharpe_variance_ddof1": 0.0002827259258371177, + "trials": [ + "momentum20", + "momentum60", + "momentum120", + "cash" + ] + }, + "with_equalweight_control": { + "dsr": 0.5750830794792382, + "expected_max_sharpe_sr0": 0.02141737891040903, + "observations": 301, + "per_trial": { + "cash": { + "kurtosis": null, + "mean": 0.0, + "n": 301, + "sharpe": 0.0, + "skew": null, + "std": 0.0 + }, + "equalweight": { + "kurtosis": 11.325886200438843, + "mean": 0.0008972410662137079, + "n": 301, + "sharpe": 0.03257184566045634, + "skew": -1.1840401587874543, + "std": 0.02754652209662771 + }, + "momentum120": { + "kurtosis": 11.70581994534825, + "mean": 0.000647213217448946, + "n": 301, + "sharpe": 0.02539620742431157, + "skew": -1.4941526859828613, + "std": 0.025484640546342932 + }, + "momentum20": { + "kurtosis": 10.534059391680314, + "mean": 0.00047619722098306353, + "n": 301, + "sharpe": 0.020820777225615553, + "skew": -0.6864816724012911, + "std": 0.022871250953936714 + }, + "momentum60": { + "kurtosis": 14.863770264503078, + "mean": -0.00022989605827723083, + "n": 301, + "sharpe": -0.00993755279683209, + "skew": -2.2018993871687806, + "std": 0.02313407163487146 + } + }, + "psr_vs_zero": 0.7098198477046372, + "selected_sharpe": 0.03257184566045634, + "selected_trial": "equalweight", + "trial_sharpe_variance_ddof1": 0.00032251312952154255, + "trials": [ + "momentum20", + "momentum60", + "momentum120", + "cash", + "equalweight" + ] + } + }, + "first_decision": "2015-01-02", + "inputs": { + "evaluate_py_sha256": "e62af085613c0b341ffdfc2a0eac94255551325f430844f29e317a26ede6cedc", + "ledger_sha256": "08ea5309466045574ba4b9c8524608ed66c91edb22cb592a5bad84e1e2f42672", + "overfitting_py_sha256": "c44a5f7d00f8cb2b9787ab8f408e76731a99e4ac7e67f69f3efbb34b4b7eebdf", + "params_sha256": "0e371da46601c673453a40ebcb608e38512348c37bf95a63ac7793d263c21487", + "research_receipt_sha256": "92a2c4c1984c7cd38af21fbc9b5290395805fed3177522b19d408ad50c59d56b" + }, + "last_decision": "2020-12-16", + "ledger_records": 6605, + "pbo": { + "momentum_only": { + "mean_S10": { + "combinations": 252, + "degradation_intercept": 0.0008984119858695838, + "degradation_slope": -0.9118725676377238, + "in_sample_selection_counts": [ + 107, + 25, + 120 + ], + "lambda_histogram": { + "-1.098612": 116, + "0.000000": 120, + "1.098612": 16 + }, + "lambda_max": 1.0986122886681098, + "lambda_mean": -0.4359572574079801, + "lambda_min": -1.0986122886681098, + "lambda_zero_count": 120, + "metric": "mean", + "partitions": 10, + "pbo": 0.9365079365079365, + "pbo_strict": 0.4603174603174603, + "probability_oos_loss": 0.5317460317460317, + "rows": 301, + "rows_dropped_earliest": 1, + "rows_per_partition": 30, + "rows_used": 300, + "selected_in_sample_mean": 0.0011109767822259436, + "selected_out_of_sample_mean": -0.00011465726512468383, + "trials": 3 + }, + "mean_S12": { + "combinations": 924, + "degradation_intercept": 0.0009573417091899579, + "degradation_slope": -0.9115018430994034, + "in_sample_selection_counts": [ + 408, + 61, + 455 + ], + "lambda_histogram": { + "-1.098612": 343, + "0.000000": 511, + "1.098612": 70 + }, + "lambda_max": 1.0986122886681098, + "lambda_mean": -0.32458999437921426, + "lambda_min": -1.0986122886681098, + "lambda_zero_count": 511, + "metric": "mean", + "partitions": 12, + "pbo": 0.9242424242424242, + "pbo_strict": 0.3712121212121212, + "probability_oos_loss": 0.48593073593073594, + "rows": 301, + "rows_dropped_earliest": 1, + "rows_per_partition": 25, + "rows_used": 300, + "selected_in_sample_mean": 0.001025869501173754, + "selected_out_of_sample_mean": 2.2259768090615506e-05, + "trials": 3 + }, + "mean_S16": { + "combinations": 12870, + "degradation_intercept": 0.0008912723722196319, + "degradation_slope": -0.9211665387182658, + "in_sample_selection_counts": [ + 5746, + 776, + 6348 + ], + "lambda_histogram": { + "-1.098612": 4428, + "0.000000": 7880, + "1.098612": 562 + }, + "lambda_max": 1.0986122886681098, + "lambda_mean": -0.33001049790139175, + "lambda_min": -1.0986122886681098, + "lambda_zero_count": 7880, + "metric": "mean", + "partitions": 16, + "pbo": 0.9563325563325563, + "pbo_strict": 0.34405594405594403, + "probability_oos_loss": 0.4966588966588967, + "rows": 301, + "rows_dropped_earliest": 13, + "rows_per_partition": 18, + "rows_used": 288, + "selected_in_sample_mean": 0.0009689287943105048, + "selected_out_of_sample_mean": -1.2724114998384463e-06, + "trials": 3 + }, + "mean_S8": { + "combinations": 70, + "degradation_intercept": 0.0009605227808458237, + "degradation_slope": -0.9140463245423784, + "in_sample_selection_counts": [ + 29, + 7, + 34 + ], + "lambda_histogram": { + "-1.098612": 25, + "0.000000": 39, + "1.098612": 6 + }, + "lambda_max": 1.0986122886681098, + "lambda_mean": -0.29819476406705836, + "lambda_min": -1.0986122886681098, + "lambda_zero_count": 39, + "metric": "mean", + "partitions": 8, + "pbo": 0.9142857142857143, + "pbo_strict": 0.35714285714285715, + "probability_oos_loss": 0.5428571428571428, + "rows": 301, + "rows_dropped_earliest": 5, + "rows_per_partition": 37, + "rows_used": 296, + "selected_in_sample_mean": 0.0009350556555339403, + "selected_out_of_sample_mean": 0.00010583859566246121, + "trials": 3 + }, + "sharpe_S10": { + "combinations": 252, + "degradation_intercept": 0.02990585230917518, + "degradation_slope": -0.9276892193206676, + "in_sample_selection_counts": [ + 101, + 27, + 124 + ], + "lambda_histogram": { + "-1.098612": 120, + "0.000000": 108, + "1.098612": 24 + }, + "lambda_max": 1.0986122886681098, + "lambda_mean": -0.41851896711166087, + "lambda_min": -1.0986122886681098, + "lambda_zero_count": 108, + "metric": "sharpe", + "partitions": 10, + "pbo": 0.9047619047619048, + "pbo_strict": 0.47619047619047616, + "probability_oos_loss": 0.5357142857142857, + "rows": 301, + "rows_dropped_earliest": 1, + "rows_per_partition": 30, + "rows_used": 300, + "selected_in_sample_mean": 0.04271119999452696, + "selected_out_of_sample_mean": -0.009716867469996438, + "trials": 3 + }, + "sharpe_S12": { + "combinations": 924, + "degradation_intercept": 0.03473384022372467, + "degradation_slope": -0.9003237991945331, + "in_sample_selection_counts": [ + 395, + 72, + 457 + ], + "lambda_histogram": { + "-1.098612": 353, + "0.000000": 477, + "1.098612": 94 + }, + "lambda_max": 1.0986122886681098, + "lambda_mean": -0.30794435364181866, + "lambda_min": -1.0986122886681098, + "lambda_zero_count": 477, + "metric": "sharpe", + "partitions": 12, + "pbo": 0.8982683982683982, + "pbo_strict": 0.38203463203463206, + "probability_oos_loss": 0.4945887445887446, + "rows": 301, + "rows_dropped_earliest": 1, + "rows_per_partition": 25, + "rows_used": 300, + "selected_in_sample_mean": 0.040848690296514785, + "selected_out_of_sample_mean": -0.0020432078161543806, + "trials": 3 + }, + "sharpe_S16": { + "combinations": 12870, + "degradation_intercept": 0.028851212592936295, + "degradation_slope": -0.8725597711548888, + "in_sample_selection_counts": [ + 5373, + 990, + 6507 + ], + "lambda_histogram": { + "-1.098612": 4800, + "0.000000": 7032, + "1.098612": 1038 + }, + "lambda_max": 1.0986122886681098, + "lambda_mean": -0.32113282284144745, + "lambda_min": -1.0986122886681098, + "lambda_zero_count": 7032, + "metric": "sharpe", + "partitions": 16, + "pbo": 0.9193473193473194, + "pbo_strict": 0.372960372960373, + "probability_oos_loss": 0.498989898989899, + "rows": 301, + "rows_dropped_earliest": 13, + "rows_per_partition": 18, + "rows_used": 288, + "selected_in_sample_mean": 0.03685917292888655, + "selected_out_of_sample_mean": -0.0033106189028514247, + "trials": 3 + }, + "sharpe_S8": { + "combinations": 70, + "degradation_intercept": 0.030083454037995377, + "degradation_slope": -0.9262724087164941, + "in_sample_selection_counts": [ + 23, + 8, + 39 + ], + "lambda_histogram": { + "-1.098612": 26, + "0.000000": 28, + "1.098612": 16 + }, + "lambda_max": 1.0986122886681098, + "lambda_mean": -0.15694461266687285, + "lambda_min": -1.0986122886681098, + "lambda_zero_count": 28, + "metric": "sharpe", + "partitions": 8, + "pbo": 0.7714285714285715, + "pbo_strict": 0.37142857142857144, + "probability_oos_loss": 0.5428571428571428, + "rows": 301, + "rows_dropped_earliest": 5, + "rows_per_partition": 37, + "rows_used": 296, + "selected_in_sample_mean": 0.03183331097631105, + "selected_out_of_sample_mean": 0.0005971364025465317, + "trials": 3 + } + }, + "primary": { + "mean_S10": { + "combinations": 252, + "degradation_intercept": 0.0003755960182084304, + "degradation_slope": -0.6974169642087382, + "in_sample_selection_counts": [ + 73, + 12, + 98, + 69 + ], + "lambda_histogram": { + "-0.405465": 88, + "-1.386294": 125, + "0.405465": 33, + "1.386294": 6 + }, + "lambda_max": 1.3862943611198908, + "lambda_mean": -0.7431333726953017, + "lambda_min": -1.3862943611198906, + "lambda_zero_count": 0, + "metric": "mean", + "partitions": 10, + "pbo": 0.8452380952380952, + "pbo_strict": 0.8452380952380952, + "probability_oos_loss": 0.5119047619047619, + "rows": 301, + "rows_dropped_earliest": 1, + "rows_per_partition": 30, + "rows_used": 300, + "selected_in_sample_mean": 0.0013061708589959147, + "selected_out_of_sample_mean": -0.0005353496970104204, + "trials": 4 + }, + "mean_S12": { + "combinations": 924, + "degradation_intercept": 0.0003975356503207397, + "degradation_slope": -0.6748833832155265, + "in_sample_selection_counts": [ + 255, + 31, + 386, + 252 + ], + "lambda_histogram": { + "-0.405465": 271, + "-1.386294": 449, + "0.405465": 182, + "1.386294": 22 + }, + "lambda_max": 1.3862943611198908, + "lambda_mean": -0.6796905701513202, + "lambda_min": -1.3862943611198906, + "lambda_zero_count": 0, + "metric": "mean", + "partitions": 12, + "pbo": 0.7792207792207793, + "pbo_strict": 0.7792207792207793, + "probability_oos_loss": 0.4816017316017316, + "rows": 301, + "rows_dropped_earliest": 1, + "rows_per_partition": 25, + "rows_used": 300, + "selected_in_sample_mean": 0.0012389532857593503, + "selected_out_of_sample_mean": -0.0004386133348185236, + "trials": 4 + }, + "mean_S16": { + "combinations": 12870, + "degradation_intercept": 0.00044423801083574936, + "degradation_slope": -0.7273275972514565, + "in_sample_selection_counts": [ + 3814, + 289, + 5490, + 3277 + ], + "lambda_histogram": { + "-0.405465": 4128, + "-1.386294": 5854, + "0.405465": 2632, + "1.386294": 256 + }, + "lambda_max": 1.3862943611198908, + "lambda_mean": -0.6501205621817375, + "lambda_min": -1.3862943611198906, + "lambda_zero_count": 0, + "metric": "mean", + "partitions": 16, + "pbo": 0.7756021756021756, + "pbo_strict": 0.7756021756021756, + "probability_oos_loss": 0.48003108003108, + "rows": 301, + "rows_dropped_earliest": 13, + "rows_per_partition": 18, + "rows_used": 288, + "selected_in_sample_mean": 0.0011880286000091778, + "selected_out_of_sample_mean": -0.00041984797627493757, + "trials": 4 + }, + "mean_S8": { + "combinations": 70, + "degradation_intercept": 0.00012216926410789342, + "degradation_slope": -0.6075829596038457, + "in_sample_selection_counts": [ + 12, + 2, + 26, + 30 + ], + "lambda_histogram": { + "-0.405465": 18, + "-1.386294": 45, + "0.405465": 7 + }, + "lambda_max": 0.4054651081081642, + "lambda_mean": -0.9549051777083554, + "lambda_min": -1.3862943611198906, + "lambda_zero_count": 0, + "metric": "mean", + "partitions": 8, + "pbo": 0.9, + "pbo_strict": 0.9, + "probability_oos_loss": 0.5142857142857142, + "rows": 301, + "rows_dropped_earliest": 5, + "rows_per_partition": 37, + "rows_used": 296, + "selected_in_sample_mean": 0.0013772289987410554, + "selected_out_of_sample_mean": -0.0007146116069994381, + "trials": 4 + }, + "sharpe_S10": { + "combinations": 252, + "degradation_intercept": 0.014650477575139365, + "degradation_slope": -0.7779968069155431, + "in_sample_selection_counts": [ + 73, + 14, + 96, + 69 + ], + "lambda_histogram": { + "-0.405465": 86, + "-1.386294": 125, + "0.405465": 37, + "1.386294": 4 + }, + "lambda_max": 1.3862943611198908, + "lambda_mean": -0.7444817777492334, + "lambda_min": -1.3862943611198906, + "lambda_zero_count": 0, + "metric": "sharpe", + "partitions": 10, + "pbo": 0.8373015873015873, + "pbo_strict": 0.8373015873015873, + "probability_oos_loss": 0.5158730158730159, + "rows": 301, + "rows_dropped_earliest": 1, + "rows_per_partition": 30, + "rows_used": 300, + "selected_in_sample_mean": 0.052143438043757, + "selected_out_of_sample_mean": -0.025916950724502032, + "trials": 4 + }, + "sharpe_S12": { + "combinations": 924, + "degradation_intercept": 0.013853123632035766, + "degradation_slope": -0.6874544874643813, + "in_sample_selection_counts": [ + 269, + 40, + 363, + 252 + ], + "lambda_histogram": { + "-0.405465": 266, + "-1.386294": 446, + "0.405465": 200, + "1.386294": 12 + }, + "lambda_max": 1.3862943611198908, + "lambda_mean": -0.6801000539623067, + "lambda_min": -1.3862943611198906, + "lambda_zero_count": 0, + "metric": "sharpe", + "partitions": 12, + "pbo": 0.7705627705627706, + "pbo_strict": 0.7705627705627706, + "probability_oos_loss": 0.4902597402597403, + "rows": 301, + "rows_dropped_earliest": 1, + "rows_per_partition": 25, + "rows_used": 300, + "selected_in_sample_mean": 0.05072668963403052, + "selected_out_of_sample_mean": -0.021019166791091427, + "trials": 4 + }, + "sharpe_S16": { + "combinations": 12870, + "degradation_intercept": 0.01674539901662133, + "degradation_slope": -0.7513857960806316, + "in_sample_selection_counts": [ + 3964, + 336, + 5293, + 3277 + ], + "lambda_histogram": { + "-0.405465": 3960, + "-1.386294": 5770, + "0.405465": 3056, + "1.386294": 84 + }, + "lambda_max": 1.3862943611198908, + "lambda_mean": -0.6409487331046992, + "lambda_min": -1.3862943611198906, + "lambda_zero_count": 0, + "metric": "sharpe", + "partitions": 16, + "pbo": 0.756021756021756, + "pbo_strict": 0.756021756021756, + "probability_oos_loss": 0.48166278166278165, + "rows": 301, + "rows_dropped_earliest": 13, + "rows_per_partition": 18, + "rows_used": 288, + "selected_in_sample_mean": 0.047414314217345, + "selected_out_of_sample_mean": -0.01888104321719565, + "trials": 4 + }, + "sharpe_S8": { + "combinations": 70, + "degradation_intercept": 0.0027391817604871457, + "degradation_slope": -0.6594005137910001, + "in_sample_selection_counts": [ + 12, + 2, + 26, + 30 + ], + "lambda_histogram": { + "-0.405465": 15, + "-1.386294": 43, + "0.405465": 12 + }, + "lambda_max": 0.4054651081081642, + "lambda_mean": -0.8689578978925684, + "lambda_min": -1.3862943611198906, + "lambda_zero_count": 0, + "metric": "sharpe", + "partitions": 8, + "pbo": 0.8285714285714286, + "pbo_strict": 0.8285714285714286, + "probability_oos_loss": 0.5142857142857142, + "rows": 301, + "rows_dropped_earliest": 5, + "rows_per_partition": 37, + "rows_used": 296, + "selected_in_sample_mean": 0.053047748257383835, + "selected_out_of_sample_mean": -0.032240530695887386, + "trials": 4 + } + }, + "with_equalweight_control": { + "mean_S10": { + "combinations": 252, + "degradation_intercept": 0.0008124346550125723, + "degradation_slope": -0.8238587597598237, + "in_sample_selection_counts": [ + 64, + 6, + 71, + 49, + 62 + ], + "lambda_histogram": { + "-0.693147": 73, + "-1.609438": 105, + "0.000000": 32, + "0.693147": 30, + "1.609438": 12 + }, + "lambda_max": 1.6094379124341007, + "lambda_mean": -0.7122343437319405, + "lambda_min": -1.6094379124341005, + "lambda_zero_count": 32, + "metric": "mean", + "partitions": 10, + "pbo": 0.8333333333333334, + "pbo_strict": 0.7063492063492064, + "probability_oos_loss": 0.47619047619047616, + "rows": 301, + "rows_dropped_earliest": 1, + "rows_per_partition": 30, + "rows_used": 300, + "selected_in_sample_mean": 0.0013663870558545004, + "selected_out_of_sample_mean": -0.00031327529017559347, + "trials": 5 + }, + "mean_S12": { + "combinations": 924, + "degradation_intercept": 0.0007711894468638225, + "degradation_slope": -0.7328454819776022, + "in_sample_selection_counts": [ + 156, + 18, + 253, + 172, + 325 + ], + "lambda_histogram": { + "-0.693147": 273, + "-1.609438": 339, + "0.000000": 185, + "0.693147": 85, + "1.609438": 42 + }, + "lambda_max": 1.6094379124341007, + "lambda_mean": -0.6583492748248891, + "lambda_min": -1.6094379124341005, + "lambda_zero_count": 185, + "metric": "mean", + "partitions": 12, + "pbo": 0.8625541125541125, + "pbo_strict": 0.6623376623376623, + "probability_oos_loss": 0.47294372294372294, + "rows": 301, + "rows_dropped_earliest": 1, + "rows_per_partition": 25, + "rows_used": 300, + "selected_in_sample_mean": 0.001423733394545966, + "selected_out_of_sample_mean": -0.0002721871388698237, + "trials": 5 + }, + "mean_S16": { + "combinations": 12870, + "degradation_intercept": 0.0007905055762317812, + "degradation_slope": -0.7793453291223932, + "in_sample_selection_counts": [ + 2543, + 120, + 3643, + 2446, + 4118 + ], + "lambda_histogram": { + "-0.693147": 3097, + "-1.609438": 4861, + "0.000000": 2751, + "0.693147": 1591, + "1.609438": 570 + }, + "lambda_max": 1.6094379124341007, + "lambda_mean": -0.6177138878149186, + "lambda_min": -1.6094379124341005, + "lambda_zero_count": 2751, + "metric": "mean", + "partitions": 16, + "pbo": 0.832090132090132, + "pbo_strict": 0.6183372183372183, + "probability_oos_loss": 0.4575757575757576, + "rows": 301, + "rows_dropped_earliest": 13, + "rows_per_partition": 18, + "rows_used": 288, + "selected_in_sample_mean": 0.0013298401978357331, + "selected_out_of_sample_mean": -0.0002458991704306967, + "trials": 5 + }, + "mean_S8": { + "combinations": 70, + "degradation_intercept": 0.00042937317639973383, + "degradation_slope": -0.6247513283072168, + "in_sample_selection_counts": [ + 8, + 0, + 12, + 25, + 25 + ], + "lambda_histogram": { + "-0.693147": 17, + "-1.609438": 36, + "0.000000": 11, + "0.693147": 6 + }, + "lambda_max": 0.6931471805599452, + "lambda_mean": -0.936634054768386, + "lambda_min": -1.6094379124341005, + "lambda_zero_count": 11, + "metric": "mean", + "partitions": 8, + "pbo": 0.9142857142857143, + "pbo_strict": 0.7571428571428571, + "probability_oos_loss": 0.4857142857142857, + "rows": 301, + "rows_dropped_earliest": 5, + "rows_per_partition": 37, + "rows_used": 296, + "selected_in_sample_mean": 0.0015279743255972446, + "selected_out_of_sample_mean": -0.0005252308131364684, + "trials": 5 + }, + "sharpe_S10": { + "combinations": 252, + "degradation_intercept": 0.028639078022273817, + "degradation_slope": -0.858701567497155, + "in_sample_selection_counts": [ + 68, + 8, + 65, + 49, + 62 + ], + "lambda_histogram": { + "-0.693147": 73, + "-1.609438": 103, + "0.000000": 39, + "0.693147": 29, + "1.609438": 8 + }, + "lambda_max": 1.6094379124341007, + "lambda_mean": -0.7277582445471316, + "lambda_min": -1.6094379124341005, + "lambda_zero_count": 39, + "metric": "sharpe", + "partitions": 10, + "pbo": 0.8531746031746031, + "pbo_strict": 0.6984126984126984, + "probability_oos_loss": 0.4722222222222222, + "rows": 301, + "rows_dropped_earliest": 1, + "rows_per_partition": 30, + "rows_used": 300, + "selected_in_sample_mean": 0.05483720380767608, + "selected_out_of_sample_mean": -0.018449714844538587, + "trials": 5 + }, + "sharpe_S12": { + "combinations": 924, + "degradation_intercept": 0.022452792840967795, + "degradation_slope": -0.6241540598620049, + "in_sample_selection_counts": [ + 180, + 26, + 227, + 172, + 319 + ], + "lambda_histogram": { + "-0.693147": 250, + "-1.609438": 338, + "0.000000": 214, + "0.693147": 78, + "1.609438": 44 + }, + "lambda_max": 1.6094379124341007, + "lambda_mean": -0.6411212784761213, + "lambda_min": -1.6094379124341005, + "lambda_zero_count": 214, + "metric": "sharpe", + "partitions": 12, + "pbo": 0.8679653679653679, + "pbo_strict": 0.6363636363636364, + "probability_oos_loss": 0.4772727272727273, + "rows": 301, + "rows_dropped_earliest": 1, + "rows_per_partition": 25, + "rows_used": 300, + "selected_in_sample_mean": 0.0588850571579611, + "selected_out_of_sample_mean": -0.014300554649379833, + "trials": 5 + }, + "sharpe_S16": { + "combinations": 12870, + "degradation_intercept": 0.02700644710335813, + "degradation_slope": -0.754633031920549, + "in_sample_selection_counts": [ + 2896, + 167, + 3542, + 2446, + 3819 + ], + "lambda_histogram": { + "-0.693147": 3167, + "-1.609438": 4757, + "0.000000": 3198, + "0.693147": 1504, + "1.609438": 244 + }, + "lambda_max": 1.6094379124341007, + "lambda_mean": -0.6539313955000999, + "lambda_min": -1.6094379124341005, + "lambda_zero_count": 3198, + "metric": "sharpe", + "partitions": 16, + "pbo": 0.8641802641802642, + "pbo_strict": 0.6156954156954157, + "probability_oos_loss": 0.46907536907536906, + "rows": 301, + "rows_dropped_earliest": 13, + "rows_per_partition": 18, + "rows_used": 288, + "selected_in_sample_mean": 0.05285136074821765, + "selected_out_of_sample_mean": -0.012876935499196043, + "trials": 5 + }, + "sharpe_S8": { + "combinations": 70, + "degradation_intercept": 0.011697305728640854, + "degradation_slope": -0.6344411567427698, + "in_sample_selection_counts": [ + 8, + 0, + 17, + 25, + 20 + ], + "lambda_histogram": { + "-0.693147": 16, + "-1.609438": 32, + "0.000000": 19, + "0.693147": 3 + }, + "lambda_max": 0.6931471805599452, + "lambda_mean": -0.8644703792167215, + "lambda_min": -1.6094379124341005, + "lambda_zero_count": 19, + "metric": "sharpe", + "partitions": 8, + "pbo": 0.9571428571428572, + "pbo_strict": 0.6857142857142857, + "probability_oos_loss": 0.4857142857142857, + "rows": 301, + "rows_dropped_earliest": 5, + "rows_per_partition": 37, + "rows_used": 296, + "selected_in_sample_mean": 0.05786714389700346, + "selected_out_of_sample_mean": -0.025015991982774335, + "trials": 5 + } + } + }, + "primary": { + "dsr": 0.5520685371554236, + "dsr_selected_trial": "momentum120", + "pbo": 0.7756021756021756, + "pbo_strict": 0.7756021756021756, + "probability_oos_loss": 0.48003108003108 + }, + "walk_forward": { + "chosen_counts": { + "cash": 7, + "momentum120": 4, + "momentum20": 4, + "momentum60": 5 + }, + "first_decision": "2016-01-14", + "folds": 20, + "last_decision": "2020-12-16", + "moments": { + "kurtosis": 17.496936247274043, + "mean": -0.0003171285205987647, + "n": 249, + "skew": -1.812947934946181, + "std": 0.02099941875059171 + }, + "observations": 249, + "psr_vs_zero": 0.40476571908946396, + "sharpe": -0.015101776118914188 + } + }, + "conclusion": [ + "DSR of the best retained selection candidate is 0.552 (momentum120), far below a 0.95 confidence threshold, even though only four trials are deflated; its undeflated PSR(0) is 0.667. Adding the equal-weight control (best: equalweight, DSR 0.575) or restricting to momentum rules (DSR 0.561) does not change this.", + "PBO is 0.776 for the primary set (S=16, mean label): the in-sample best candidate ranks below the out-of-sample median in about three of four CSCV splits (N=4, so no split lands exactly on the median), and the selected candidate's out-of-sample mean regresses negatively on its in-sample mean (slope -0.727). Every sensitivity cell (S 8 to 16, mean or Sharpe metric, three trial sets) has PBO (lambda <= 0) between 0.76 and 0.96; the strict lambda < 0 share is lower where an odd trial count allows exact median ties (momentum-only N=3: 0.34 to 0.48; with the control N=5: 0.62 to 0.76).", + "The frozen walk-forward selector's development out-of-sample series (249 episodes over 20 folds) has negative mean and Sharpe -0.0151, PSR(0) 0.405. This is consistent with the research-evaluation README, which makes no persistent-advantage claim for the training-selected rule.", + "Conclusion: over these retained candidates the evidence is consistent with selection noise (or mild anti-persistence of in-sample ranking), not skill. This is a negative diagnostic for this three-ETF control lane; it neither tunes, promotes nor rejects any future strategy." + ], + "limitations": [ + "Inputs are raw-price five-session labels less a fixed 20bp proxy, not portfolio returns; the Sharpe ratios are per-episode label ratios, not investable or annualized Sharpe ratios.", + "The trial count is only this lane's frozen candidate set (4, sensitivity 3 and 5). Trials run elsewhere in the research program are not counted, so the DSR here is an optimistic upper bound relative to program-wide multiple testing; no effective-N clustering was applied although momentum60 and momentum120 share 60% of development labels.", + "CSCV with three to five highly correlated trials gives only a few distinct relative ranks (four logit values in the primary S=16 run) and assumes the contiguous 18-episode blocks are exchangeable; PBO is descriptive and carries no p-value.", + "Only development episodes (2015-01-02 to 2020-12-16) are used; the inspected 2021Q1 reserved segment (11 episodes) is excluded and was not re-scored.", + "The earliest T mod S rows (13 at S=16) are dropped; results for other truncation choices were not computed.", + "The DSR worked-example constants are transcribed from the paper rather than re-fetched; the PBO tests are locally constructed synthetic fixtures, not upstream tests.", + "Evidence class is a local integration check with an independent numpy recomputation; no upstream implementation was executed or compared." + ] +} diff --git a/blueprints/us-equities/research-evaluation/README.md b/blueprints/us-equities/research-evaluation/README.md index 95dfdddb8..6b2d6a767 100644 --- a/blueprints/us-equities/research-evaluation/README.md +++ b/blueprints/us-equities/research-evaluation/README.md @@ -76,6 +76,15 @@ Independent review verified all 42 retained artifact hashes, exact public/native result equality, all 21 selection hashes, the pre-scoring plan/source/lock freeze, the complete ledger counts and 12 focused tests without rerunning scoring. +## Overfitting controls — September 24, 2026 + +The [overfitting-controls lane](../overfitting-controls/README.md) computed two +diagnostics offline from this run's retained 6,605-record ledger over 301 +development episodes. For the four selection candidates, the deflated Sharpe +ratio is **0.552** and the CSCV probability of backtest overfitting is +**0.776**. The diagnostics add no skill evidence and change no selection. The +reserved segment was not re-scored. + ## Native adoption and execution Keep the working SDK unchanged. Choose a fresh private `WAVE_DIR`, an installed diff --git a/manifests/evidence.json b/manifests/evidence.json index 193ace526..f1b3a8073 100644 --- a/manifests/evidence.json +++ b/manifests/evidence.json @@ -8818,8 +8818,8 @@ }, { "path": "blueprints/us-equities/research-evaluation/README.md", - "sha256": "5f6e4b5cc7338e9f5fc3edeb0d2d5092f1c2ba779d0243ed3ba29da76a80edde", - "bytes": 8589 + "sha256": "89b428b4ea8c289472805bb2eb5c45d7dcb0333011ad61097fb0d6bf5aee9d5c", + "bytes": 9054 }, { "path": "blueprints/us-equities/research-evaluation/evaluate.py", diff --git a/tests/test_overfitting_controls.py b/tests/test_overfitting_controls.py new file mode 100644 index 000000000..1a2bee0d6 --- /dev/null +++ b/tests/test_overfitting_controls.py @@ -0,0 +1,183 @@ +"""Deflated Sharpe ratio and CSCV PBO: published example, synthetic fixtures and receipt replay.""" +import importlib.util +import itertools +import json +import math +from pathlib import Path +import random +import statistics +import unittest + +ROOT = Path(__file__).resolve().parents[1] +HERE = ROOT / "blueprints/us-equities/overfitting-controls" + + +def load(name, path): + spec = importlib.util.spec_from_file_location(name, path) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +of = load("overfitting_controls_under_test", HERE / "overfitting.py") + + +def brute_force_pbo(matrix, partitions, metric): + """Independent direct recomputation: concatenate rows, statistics module, explicit ranks.""" + t = len(matrix) + size = t // partitions + used = matrix[t - size * partitions:] + blocks = [used[b * size:(b + 1) * size] for b in range(partitions)] + n = len(matrix[0]) + + def score(rows, c): + col = [r[c] for r in rows] + mean = statistics.fmean(col) + if metric == "mean": + return mean + std = statistics.pstdev(col) + return 0.0 if std == 0 else mean / std + + logits = [] + for chosen in itertools.combinations(range(partitions), partitions // 2): + train = [r for b in chosen for r in blocks[b]] + test = [r for b in range(partitions) if b not in chosen for r in blocks[b]] + is_scores = [score(train, c) for c in range(n)] + oos = [score(test, c) for c in range(n)] + best = max(range(n), key=lambda c: (is_scores[c], -c)) + below = sum(v < oos[best] for v in oos) + equal = sum(v == oos[best] for v in oos) + rank = below + (equal + 1) / 2.0 + omega = rank / (n + 1) + logits.append(math.log(omega / (1 - omega))) + return logits + + +class DeflatedSharpeTests(unittest.TestCase): + def test_published_numerical_example(self): + # Bailey and Lopez de Prado (2014), numerical example: N = 100 independent + # trials, annualized V[SR_n] = 1/2, selected annualized SR = 2.5, T = 1250 + # daily observations (250 per year), skewness -3, kurtosis 10. The paper + # reports SR0 ~= 0.1132 (non-annualized) and DSR ~= 0.9004. + sr0_annual = of.expected_max_sharpe(100, 0.5) + sr0 = sr0_annual / math.sqrt(250) + self.assertAlmostEqual(sr0, 0.1132, places=4) + dsr, benchmark = of.deflated_sharpe(2.5 / math.sqrt(250), 1250, -3.0, 10.0, 100, 0.5 / 250) + self.assertAlmostEqual(benchmark, sr0, places=12) + self.assertAlmostEqual(dsr, 0.9004, places=4) + + def test_psr_reduces_to_normal_case(self): + sr, t = 0.1, 500 + self.assertAlmostEqual(of.probabilistic_sharpe(sr, t, 0.0, 3.0), + statistics.NormalDist().cdf(sr * math.sqrt(t - 1) / math.sqrt(1 + sr * sr / 2)), places=14) + self.assertEqual(of.probabilistic_sharpe(0.2, 100, -1.0, 6.0, 0.2), 0.5) + + def test_negative_skew_and_fat_tails_lower_psr(self): + normal = of.probabilistic_sharpe(0.1, 250, 0.0, 3.0) + self.assertLess(of.probabilistic_sharpe(0.1, 250, -2.0, 3.0), normal) + self.assertLess(of.probabilistic_sharpe(0.1, 250, 0.0, 12.0), normal) + + def test_expected_max_matches_order_statistic_tables(self): + # Expected maximum of N iid standard normals (Harter 1961): N=100 -> 2.50759. + # The paper's approximation is asymptotic; allow 1.5% at N=100. + self.assertAlmostEqual(of.expected_max_sharpe(100, 1.0) / 2.50759, 1.0, delta=0.015) + self.assertEqual(of.expected_max_sharpe(1, 4.0), 0.0) + self.assertGreater(of.expected_max_sharpe(1000, 1.0), of.expected_max_sharpe(100, 1.0)) + with self.assertRaises(ValueError): + of.expected_max_sharpe(0, 1.0) + + def test_more_trials_deflate_more(self): + first, _ = of.deflated_sharpe(0.1, 500, 0.0, 3.0, 2, 0.01) + second, _ = of.deflated_sharpe(0.1, 500, 0.0, 3.0, 50, 0.01) + self.assertLess(second, first) + + def test_moments_and_degenerate_series(self): + m = of.moments([1.0, 2.0, 3.0, 4.0]) + self.assertAlmostEqual(m["std"], statistics.pstdev([1, 2, 3, 4]), places=14) + self.assertAlmostEqual(m["skew"], 0.0, places=14) + self.assertAlmostEqual(m["kurtosis"], 1.64, places=12) + self.assertEqual(of.sharpe([0.0, 0.0, 0.0]), 0.0) + with self.assertRaises(ValueError): + of.sharpe([1.0, 1.0]) + with self.assertRaises(ValueError): + of.moments([1.0]) + + +class PboTests(unittest.TestCase): + def test_reversal_is_fully_overfit(self): + # Two blocks: trial 0 wins block 0, trial 1 wins block 1. Whichever block is + # in-sample, the in-sample winner is last out of sample. + matrix = [[1.0, 0.0], [1.0, 0.0], [0.0, 1.0], [0.0, 1.0]] + result = of.cscv_pbo(matrix, partitions=2) + self.assertEqual(result["combinations"], 2) + self.assertEqual(result["pbo"], 1.0) + self.assertAlmostEqual(result["lambda_max"], math.log(0.5), places=14) + + def test_dominant_trial_is_not_overfit(self): + rng = random.Random(7) + matrix = [[rng.gauss(0, 1) + 10.0] + [rng.gauss(0, 1) for _ in range(5)] for _ in range(160)] + result = of.cscv_pbo(matrix, partitions=8) + self.assertEqual(result["combinations"], math.comb(8, 4)) + self.assertEqual(result["pbo"], 0.0) + self.assertEqual(result["in_sample_selection_counts"][0], result["combinations"]) + self.assertEqual(result["probability_oos_loss"], 0.0) + + def test_pure_noise_is_near_one_half(self): + pbos = [] + for seed in range(20): + rng = random.Random(seed) + matrix = [[rng.gauss(0, 1) for _ in range(10)] for _ in range(200)] + pbos.append(of.cscv_pbo(matrix, partitions=10)["pbo"]) + self.assertAlmostEqual(statistics.fmean(pbos), 0.5, delta=0.1) + + def test_block_merge_equals_direct_recomputation(self): + rng = random.Random(11) + matrix = [[rng.gauss(0.001 * c, 0.02) for c in range(4)] + [0.0] for _ in range(103)] + for metric in ("mean", "sharpe"): + fast = of.cscv_pbo(matrix, partitions=8, metric=metric) + slow = brute_force_pbo(matrix, 8, metric) + self.assertEqual(fast["rows_dropped_earliest"], 103 - 8 * 12) + self.assertEqual(len(slow), fast["combinations"]) + for a, b in zip(fast["logits"], slow): + self.assertAlmostEqual(a, b, places=12) + self.assertEqual(fast["pbo"], sum(x <= 0 for x in slow) / len(slow)) + + def test_ties_use_average_rank_and_first_in_sample(self): + matrix = [[0.0, 0.0, 0.0] for _ in range(8)] + result = of.cscv_pbo(matrix, partitions=4) + self.assertEqual(result["in_sample_selection_counts"], [6, 0, 0]) + self.assertEqual(result["lambda_zero_count"], 6) + self.assertEqual(result["pbo"], 1.0) + self.assertEqual(result["pbo_strict"], 0.0) + + def test_rejects_invalid_configuration(self): + for args in ([[1.0, 2.0]] * 8, 3), ([[1.0]] * 8, 2), ([[1.0, 2.0]] * 3, 2), ([[1.0, float("nan")]] * 8, 2): + with self.assertRaises(ValueError): + of.cscv_pbo(*args) + + +class RetainedLedgerReplayTests(unittest.TestCase): + def test_receipt_results_replay_from_retained_ledger(self): + receipt = json.loads((HERE / "receipt.json").read_text()) + runner = load("overfitting_controls_runner", HERE / "evaluate.py") + replay = runner.compute() + + def close(a, b, where): + if isinstance(a, dict): + self.assertEqual(sorted(a), sorted(b), where) + for k in a: + close(a[k], b[k], where + "/" + k) + elif isinstance(a, list): + self.assertEqual(len(a), len(b), where) + for i, (x, y) in enumerate(zip(a, b)): + close(x, y, "%s/%d" % (where, i)) + elif isinstance(a, float) and isinstance(b, float): + self.assertTrue(math.isclose(a, b, rel_tol=1e-9, abs_tol=1e-12), where) + else: + self.assertEqual(a, b, where) + + close(replay, receipt["results"], "results") + + +if __name__ == "__main__": + unittest.main() From 30103cd668bf8535a149b2c47210a07c5da83b14 Mon Sep 17 00:00:00 2001 From: Scout Date: Thu, 24 Sep 2026 12:33:45 -0400 Subject: [PATCH 2/2] Stop citing the CSCV degradation slope as evidence of anti-persistence With the mean metric and equal complementary halves, each trial's in-sample mean plus its out-of-sample mean equals twice its mean over the used rows. Every split that selects the same trial therefore lies on a line of slope exactly -1, and momentum120 is selected in 5,490 of 12,870 splits, so the pooled -0.727 slope is mostly a construction identity and says little about ranking persistence. The identity is now stated in the cscv_pbo docstring, the README and the receipt limitations. The slope row is removed from the README result table, and the receipt conclusion no longer cites it. The anti-persistence wording now rests on PBO 0.776 against a no-skill level of about 0.5 for four exchangeable trials, and is marked descriptive because the splits overlap and PBO has no p-value. A new unit test checks the identity: slope exactly -1 for an always-selected trial, and selected in- plus out-of-sample means equal to twice the selection-weighted full means. The numbers did not change. evaluate.py was re-run twice on CPython 3.13.15 and once on the pinned 3.12.3 runtime, giving byte-identical stdout that differs from the original only in the overfitting.py source hash. The receipt keeps runs 1-3 and records the fix-round runs separately. Focused tests: 14 passed, 0 skipped; with research-evaluation, 26 passed, 0 skipped on 3.12.3. Full suite: 4032 OK, 490 skipped on system python3. Co-Authored-By: Claude Opus 5.5 (1M context) --- .../overfitting-controls/README.md | 19 +++++++++--- .../overfitting-controls/overfitting.py | 11 +++++++ .../overfitting-controls/receipt.json | 31 ++++++++++++++----- tests/test_overfitting_controls.py | 21 +++++++++++++ 4 files changed, 69 insertions(+), 13 deletions(-) diff --git a/blueprints/us-equities/overfitting-controls/README.md b/blueprints/us-equities/overfitting-controls/README.md index 839d7e269..5db7ebfb7 100644 --- a/blueprints/us-equities/overfitting-controls/README.md +++ b/blueprints/us-equities/overfitting-controls/README.md @@ -37,7 +37,6 @@ excluded. | **Deflated Sharpe ratio** | **0.552** | | **PBO** (S = 16, mean-label metric, 12,870 splits) | **0.776** | | Probability that the selected trial loses out of sample | 0.480 | -| Out-of-sample on in-sample degradation slope | -0.727 | | Walk-forward selected series: Sharpe / PSR(0), 249 episodes | -0.0151 / 0.405 | All sensitivity cells give PBO (lambda <= 0) between 0.76 and 0.96. The cells @@ -47,12 +46,22 @@ DSR stays between 0.55 and 0.58. The [receipt](receipt.json) retains every cell, the strict lambda < 0 shares, input and source hashes, commands and the independent recomputation. +The receipt also keeps the paper's degradation slope (the selected trial's +out-of-sample metric regressed on its in-sample metric; -0.727 in the primary +cell). It is not used as evidence. With the mean metric and equal complementary +halves, each trial's in-sample mean plus its out-of-sample mean equals twice its +mean over all used rows. Every split that selects the same trial therefore lies +on a line of slope exactly -1, and momentum120 is selected in 5,490 of the +12,870 splits. The pooled slope is pulled towards -1 by construction, and it +says almost nothing about whether rankings persist. + **Conclusion.** These retained candidates show no deflated-Sharpe evidence of skill. The in-sample best candidate usually ranks below the out-of-sample -median. The evidence is consistent with selection noise, or mild -anti-persistence of in-sample ranking, not skill. This negative diagnostic -covers only this three-ETF control lane. It does not tune, promote or reject any -future strategy. +median. With four trials and no skill, PBO would be about 0.5, so 0.776 is +consistent with selection noise or mild anti-persistence of in-sample ranking, +not skill. The CSCV splits overlap heavily and PBO has no p-value, so this +reading is descriptive. This negative diagnostic covers only this three-ETF +control lane. It does not tune, promote or reject any future strategy. ## Boundaries diff --git a/blueprints/us-equities/overfitting-controls/overfitting.py b/blueprints/us-equities/overfitting-controls/overfitting.py index bea15b47a..d449ca176 100644 --- a/blueprints/us-equities/overfitting-controls/overfitting.py +++ b/blueprints/us-equities/overfitting-controls/overfitting.py @@ -136,6 +136,17 @@ def cscv_pbo(matrix, partitions=16, metric="mean"): sample): n* = argmax in-sample metric (ties by column order), omega = average ascending out-of-sample rank of n* / (N + 1), lambda = logit(omega). PBO is the share of combinations with lambda <= 0. + + degradation_slope is the paper's pooled OLS slope of the selected trial's + out-of-sample metric on its in-sample metric. It is not evidence of ranking + persistence here. With metric="mean" and equal complementary halves, every + trial satisfies in_sample + out_of_sample = 2 * (its mean over the used rows), + so all splits that select the same trial lie on a line of slope exactly -1. + The pooled slope is therefore mostly this identity, mixed with the spread + between the full-sample means of the trials that get selected. With + metric="sharpe" the identity is not exact, but the complementary halves still + induce the same negative within-trial dependence. Judge persistence by PBO + against its no-skill level (about 0.5 for continuous ranks), not by the slope. """ rows = [[float(x) for x in row] for row in matrix] if not rows or any(len(r) != len(rows[0]) for r in rows): diff --git a/blueprints/us-equities/overfitting-controls/receipt.json b/blueprints/us-equities/overfitting-controls/receipt.json index 07aa87680..405845a46 100644 --- a/blueprints/us-equities/overfitting-controls/receipt.json +++ b/blueprints/us-equities/overfitting-controls/receipt.json @@ -54,7 +54,7 @@ "implementation": { "overfitting.py": { "path": "blueprints/us-equities/overfitting-controls/overfitting.py", - "sha256": "c44a5f7d00f8cb2b9787ab8f408e76731a99e4ac7e67f69f3efbb34b4b7eebdf" + "sha256": "1747e058c0a5830db4ae43f4101cdd4a6825422f4b5b386fd8eb7e51f803811a" }, "evaluate.py": { "path": "blueprints/us-equities/overfitting-controls/evaluate.py", @@ -101,9 +101,23 @@ "exit": 0, "stderr_bytes": 0, "stdout_sha256": "3815782f85fcb7c8aebd6defce82ae9540039f6bf408a82a83ff47dd0a83ff43" + }, + { + "id": "fix-round-runs", + "argv": [ + "python3", + "blueprints/us-equities/overfitting-controls/evaluate.py" + ], + "cwd": "checkout root", + "runtime": "CPython 3.13.15 (two runs) and CPython 3.12.3 pinned adaptive-paper runtime (one run)", + "started_utc": "2026-09-24T16:22:00Z", + "exit": 0, + "stderr_bytes": 0, + "stdout_sha256": "6bacc72f585348e624a63826ce1b65d30773fd8c50fe02e7357ebf9ceea6dbe5", + "note": "Re-run after the review fix that documented the degradation-slope identity in the overfitting.py docstring. All three stdouts are byte-identical. Only results.inputs.overfitting_py_sha256 differs from runs 1-3; every numeric result is unchanged. Runs 1-3 are kept as the original observation; their stdout hash predates the docstring change." } ], - "determinism": "All three runs produced byte-identical stdout; the results object below is that stdout.", + "determinism": "Runs 1-3 produced byte-identical stdout (3815782f...). After the docstring-only fix, three further runs (fix-round-runs) produced byte-identical stdout (6bacc72f...), differing from the original only in the overfitting.py source hash. The results object below is the fix-round stdout.", "independent_observation": { "method": "crosscheck.py recomputes the primary DSR and PBO with numpy 2.5.3 (array moments, direct row slicing of all 12870 CSCV combinations, explicit average ranks, math.erf CDF), sharing only statistics.NormalDist.inv_cdf and the ledger with the module under test.", "runtime": "CPython 3.12.3 pinned adaptive-paper runtime, exit 0, empty stderr", @@ -126,12 +140,12 @@ "classes": { "published_worked_example": "Bailey and Lopez de Prado (2014) numerical example (N=100, annualized V=1/2, SR=2.5, T=1250, skew -3, kurtosis 10): SR0 0.1132 and DSR 0.9004 reproduced to four decimals. Expected values are transcribed from the paper; the paper text was not re-fetched in this offline session.", "reference_value": "Expected maximum of 100 iid standard normals (order-statistic tables, 2.50759) within the approximation's 1.5% tolerance.", - "synthetic_fixture": "Locally constructed PBO cases: exact reversal (PBO 1), dominant trial (PBO 0), pure-noise mean near 0.5 over 20 seeds, block-merge equality with a brute-force recomputation, tie handling and invalid configurations. No published PBO numerical example with exact expected values was available offline.", + "synthetic_fixture": "Locally constructed PBO cases: exact reversal (PBO 1), dominant trial (PBO 0), pure-noise mean near 0.5 over 20 seeds, block-merge equality with a brute-force recomputation, tie handling, the mean-metric degradation-slope identity (slope exactly -1 for an always-selected trial; the selected in-sample plus out-of-sample means equal twice the selection-weighted full means) and invalid configurations. No published PBO numerical example with exact expected values was available offline.", "replay": "Recomputes this receipt's results object from the retained ledger (float tolerance 1e-9 relative)." }, "observed": { - "focused": "13 tests passed, 0 skipped on CPython 3.13.15; with tests.test_research_evaluation, 25 passed, 0 skipped on CPython 3.12.3 (pinned runtime)", - "full_suite": "python3 -m unittest on CPython 3.13.15 (system interpreter without numpy, jsonschema or PyYAML): 4031 tests OK, 490 skipped; none of the skips are in tests.test_overfitting_controls" + "focused": "After the fix round: 14 tests passed, 0 skipped on CPython 3.13.15; with tests.test_research_evaluation, 26 passed, 0 skipped on CPython 3.12.3 (pinned runtime)", + "full_suite": "python3 -m unittest on CPython 3.13.15 (system interpreter without numpy, jsonschema or PyYAML): 4032 tests OK, 490 skipped; none of the skips are in tests.test_overfitting_controls" } }, "results": { @@ -289,7 +303,7 @@ "inputs": { "evaluate_py_sha256": "e62af085613c0b341ffdfc2a0eac94255551325f430844f29e317a26ede6cedc", "ledger_sha256": "08ea5309466045574ba4b9c8524608ed66c91edb22cb592a5bad84e1e2f42672", - "overfitting_py_sha256": "c44a5f7d00f8cb2b9787ab8f408e76731a99e4ac7e67f69f3efbb34b4b7eebdf", + "overfitting_py_sha256": "1747e058c0a5830db4ae43f4101cdd4a6825422f4b5b386fd8eb7e51f803811a", "params_sha256": "0e371da46601c673453a40ebcb608e38512348c37bf95a63ac7793d263c21487", "research_receipt_sha256": "92a2c4c1984c7cd38af21fbc9b5290395805fed3177522b19d408ad50c59d56b" }, @@ -1122,14 +1136,15 @@ }, "conclusion": [ "DSR of the best retained selection candidate is 0.552 (momentum120), far below a 0.95 confidence threshold, even though only four trials are deflated; its undeflated PSR(0) is 0.667. Adding the equal-weight control (best: equalweight, DSR 0.575) or restricting to momentum rules (DSR 0.561) does not change this.", - "PBO is 0.776 for the primary set (S=16, mean label): the in-sample best candidate ranks below the out-of-sample median in about three of four CSCV splits (N=4, so no split lands exactly on the median), and the selected candidate's out-of-sample mean regresses negatively on its in-sample mean (slope -0.727). Every sensitivity cell (S 8 to 16, mean or Sharpe metric, three trial sets) has PBO (lambda <= 0) between 0.76 and 0.96; the strict lambda < 0 share is lower where an odd trial count allows exact median ties (momentum-only N=3: 0.34 to 0.48; with the control N=5: 0.62 to 0.76).", + "PBO is 0.776 for the primary set (S=16, mean label): the in-sample best candidate ranks below the out-of-sample median in about three of four CSCV splits (N=4, so no split lands exactly on the median). Every sensitivity cell (S 8 to 16, mean or Sharpe metric, three trial sets) has PBO (lambda <= 0) between 0.76 and 0.96; the strict lambda < 0 share is lower where an odd trial count allows exact median ties (momentum-only N=3: 0.34 to 0.48; with the control N=5: 0.62 to 0.76). The retained degradation_slope values (primary -0.727) are not used as evidence: with the mean metric and equal complementary halves, a trial's in-sample plus out-of-sample mean equals twice its mean over the used rows, so splits selecting the same trial lie on a slope of exactly -1 (momentum120 is selected in 5490 of 12870 splits).", "The frozen walk-forward selector's development out-of-sample series (249 episodes over 20 folds) has negative mean and Sharpe -0.0151, PSR(0) 0.405. This is consistent with the research-evaluation README, which makes no persistent-advantage claim for the training-selected rule.", - "Conclusion: over these retained candidates the evidence is consistent with selection noise (or mild anti-persistence of in-sample ranking), not skill. This is a negative diagnostic for this three-ETF control lane; it neither tunes, promotes nor rejects any future strategy." + "Conclusion: over these retained candidates the evidence is consistent with selection noise, not skill. The no-skill PBO for four exchangeable trials is about 0.5, so a PBO of 0.776 is also consistent with mild anti-persistence of in-sample ranking; the CSCV splits overlap heavily and PBO carries no p-value, so this reading is descriptive. This is a negative diagnostic for this three-ETF control lane; it neither tunes, promotes nor rejects any future strategy." ], "limitations": [ "Inputs are raw-price five-session labels less a fixed 20bp proxy, not portfolio returns; the Sharpe ratios are per-episode label ratios, not investable or annualized Sharpe ratios.", "The trial count is only this lane's frozen candidate set (4, sensitivity 3 and 5). Trials run elsewhere in the research program are not counted, so the DSR here is an optimistic upper bound relative to program-wide multiple testing; no effective-N clustering was applied although momentum60 and momentum120 share 60% of development labels.", "CSCV with three to five highly correlated trials gives only a few distinct relative ranks (four logit values in the primary S=16 run) and assumes the contiguous 18-episode blocks are exchangeable; PBO is descriptive and carries no p-value.", + "The CSCV degradation slope is dominated by a construction identity: with the mean metric and equal complementary halves, in-sample + out-of-sample = 2 x the trial's mean over the used rows, so within one selected trial the slope is exactly -1 and the pooled slope mixes that with the spread of the selected trials' full-sample means. It is retained for completeness and is not evidence about ranking persistence; no between-trial decomposition was computed. With the Sharpe metric the identity is not exact, but the same complementary-half dependence applies.", "Only development episodes (2015-01-02 to 2020-12-16) are used; the inspected 2021Q1 reserved segment (11 episodes) is excluded and was not re-scored.", "The earliest T mod S rows (13 at S=16) are dropped; results for other truncation choices were not computed.", "The DSR worked-example constants are transcribed from the paper rather than re-fetched; the PBO tests are locally constructed synthetic fixtures, not upstream tests.", diff --git a/tests/test_overfitting_controls.py b/tests/test_overfitting_controls.py index 1a2bee0d6..d78d7d4c3 100644 --- a/tests/test_overfitting_controls.py +++ b/tests/test_overfitting_controls.py @@ -150,6 +150,27 @@ def test_ties_use_average_rank_and_first_in_sample(self): self.assertEqual(result["pbo"], 1.0) self.assertEqual(result["pbo_strict"], 0.0) + def test_mean_metric_degradation_slope_is_an_identity(self): + # With the mean metric and equal complementary halves, in-sample + out-of-sample + # equals twice the trial's mean over the used rows, so a trial that is always + # selected gives a slope of exactly -1 whatever its skill. + rng = random.Random(7) + matrix = [[rng.gauss(0, 1) + 10.0] + [rng.gauss(0, 1) for _ in range(5)] for _ in range(160)] + result = of.cscv_pbo(matrix, partitions=8) + self.assertEqual(result["pbo"], 0.0) + self.assertAlmostEqual(result["degradation_slope"], -1.0, places=12) + # Mixed selection: the selected in- and out-of-sample means still sum to twice + # the selection-weighted full means. + rng = random.Random(3) + matrix = [[rng.gauss(0.001 * c, 0.02) for c in range(4)] for _ in range(103)] + result = of.cscv_pbo(matrix, partitions=8) + used = matrix[result["rows_dropped_earliest"]:] + full = [statistics.fmean(r[c] for r in used) for c in range(4)] + weighted = math.fsum(k * f for k, f in zip(result["in_sample_selection_counts"], full)) / result["combinations"] + self.assertGreater(sum(k > 0 for k in result["in_sample_selection_counts"]), 1) + self.assertAlmostEqual(result["selected_in_sample_mean"] + result["selected_out_of_sample_mean"], + 2.0 * weighted, places=14) + def test_rejects_invalid_configuration(self): for args in ([[1.0, 2.0]] * 8, 3), ([[1.0]] * 8, 2), ([[1.0, 2.0]] * 3, 2), ([[1.0, float("nan")]] * 8, 2): with self.assertRaises(ValueError):