diff --git a/blueprints/us-equities/overfitting-controls/README.md b/blueprints/us-equities/overfitting-controls/README.md new file mode 100644 index 000000000..5db7ebfb7 --- /dev/null +++ b/blueprints/us-equities/overfitting-controls/README.md @@ -0,0 +1,96 @@ +# Backtest-overfitting controls + +This lane adds two selection-bias diagnostics to the multi-candidate research +lane and applies them to the retained +[research-evaluation](../research-evaluation/README.md) candidate ledger: + +- **Deflated Sharpe ratio (DSR)**, Bailey and Lopez de Prado (2014): the + probabilistic Sharpe ratio of the best trial, measured against the expected + maximum Sharpe of N null trials rather than zero. +- **Probability of backtest overfitting (PBO)** by combinatorially symmetric + cross-validation (CSCV), Bailey, Borwein, Lopez de Prado and Zhu (2017): the + share of in-sample/out-of-sample block splits in which the in-sample best trial + ranks at or below the out-of-sample median. + +[`overfitting.py`](overfitting.py) implements both from the papers using only the +Python standard library. Nothing was copied or vendored. The citations and the +formulas are in its docstring. [`evaluate.py`](evaluate.py) applies the module +offline to the hash-pinned ledger using the parameters in +[`params.json`](params.json). Those parameters were frozen before the first +computation. [`crosscheck.py`](crosscheck.py) recomputes the primary DSR and PBO +independently with numpy. + +## Result — September 24, 2026 + +Input: the 6,605-record candidate ledger (SHA-256 `08ea5309…2f42672`, equal to +the research-evaluation receipt's `ledger_sha256`). The in-tree copy is the +byte-identical gap-wave2 regeneration. No data was refetched. After +deduplication, the development segment has 301 aligned five-session episodes per +candidate, from 2015-01-02 to 2020-12-16. The reserved 2021Q1 segment is +excluded. + +| Diagnostic (frozen four-candidate selection set) | Value | +| --- | ---: | +| Best per-episode Sharpe (momentum120), not annualized | 0.0254 | +| Expected maximum null Sharpe SR0 (N = 4) | 0.0177 | +| PSR against zero | 0.667 | +| **Deflated Sharpe ratio** | **0.552** | +| **PBO** (S = 16, mean-label metric, 12,870 splits) | **0.776** | +| Probability that the selected trial loses out of sample | 0.480 | +| Walk-forward selected series: Sharpe / PSR(0), 249 episodes | -0.0151 / 0.405 | + +All sensitivity cells give PBO (lambda <= 0) between 0.76 and 0.96. The cells +cover S = 8, 10, 12 and 16, the mean and Sharpe metrics, and three trial sets: +the primary four, adding the equal-weight control, and momentum rules only. +DSR stays between 0.55 and 0.58. The [receipt](receipt.json) retains every cell, +the strict lambda < 0 shares, input and source hashes, commands and the +independent recomputation. + +The receipt also keeps the paper's degradation slope (the selected trial's +out-of-sample metric regressed on its in-sample metric; -0.727 in the primary +cell). It is not used as evidence. With the mean metric and equal complementary +halves, each trial's in-sample mean plus its out-of-sample mean equals twice its +mean over all used rows. Every split that selects the same trial therefore lies +on a line of slope exactly -1, and momentum120 is selected in 5,490 of the +12,870 splits. The pooled slope is pulled towards -1 by construction, and it +says almost nothing about whether rankings persist. + +**Conclusion.** These retained candidates show no deflated-Sharpe evidence of +skill. The in-sample best candidate usually ranks below the out-of-sample +median. With four trials and no skill, PBO would be about 0.5, so 0.776 is +consistent with selection noise or mild anti-persistence of in-sample ranking, +not skill. The CSCV splits overlap heavily and PBO has no p-value, so this +reading is descriptive. This negative diagnostic covers only this three-ETF +control lane. It does not tune, promote or reject any future strategy. + +## Boundaries + +- The inputs are raw-price five-session labels less a fixed 20bp proxy, not + portfolio returns. The Sharpe values are per-episode label ratios. +- N counts only this lane's frozen candidates. The research program ran other + trials, so relative to all of them this DSR is optimistic. No effective-N + clustering was applied. momentum60 and momentum120 share about 60% of their + development labels. +- With three to five correlated trials, CSCV has only a few distinct ranks. + Contiguous 18-episode blocks are assumed exchangeable. PBO is descriptive and + has no p-value. +- Evidence class: a local integration check with an independent numpy + recomputation. The DSR unit test reproduces the paper's numerical example + (SR0 0.1132, DSR 0.9004). Those constants are transcribed from the paper, + which was not re-fetched offline. The PBO tests are locally constructed + synthetic fixtures. No upstream implementation was executed. + +## Reproduce + +```sh +python3 blueprints/us-equities/overfitting-controls/evaluate.py +python3 -m unittest tests.test_overfitting_controls -v +# optional independent recomputation (needs numpy) +python blueprints/us-equities/overfitting-controls/crosscheck.py +``` + +Reusing the module for another candidate ledger: pass a T x N matrix of +per-period trial performance to `cscv_pbo`. Pass the selected trial's Sharpe, +moments, trial count and cross-trial Sharpe variance to `deflated_sharpe`. +Freeze the trial set and the parameters before looking at the results. Count +every trial that was actually tried. diff --git a/blueprints/us-equities/overfitting-controls/crosscheck.py b/blueprints/us-equities/overfitting-controls/crosscheck.py new file mode 100644 index 000000000..a95637afc --- /dev/null +++ b/blueprints/us-equities/overfitting-controls/crosscheck.py @@ -0,0 +1,43 @@ +"""Independent recomputation with numpy (different code path from overfitting.py); normal quantiles from statistics.NormalDist, CDF from math.erf.""" +import json, sys, itertools, math +from decimal import Decimal +import numpy as np +from statistics import NormalDist +ppf = NormalDist().inv_cdf +cdf = lambda z: 0.5 * (1 + math.erf(z / math.sqrt(2))) + +def rankdata(a): + a = np.asarray(a); r = np.empty(len(a)) + for i, x in enumerate(a): + r[i] = (a < x).sum() + ((a == x).sum() + 1) / 2 + return r + +root = sys.argv[1] if len(sys.argv) > 1 else "." +ledger = json.load(open(root + "/evidence/artifacts/gap-wave2-20260923/us-equities__portfolio-risk/raw/3/regenerated/run-1/candidate-ledger.json")) +trials = ["momentum20", "momentum60", "momentum120", "cash"] +v = {} +for r in ledger: + if r["status"] == "observed" and r["decision"] <= "2020-12-31": + v[(r["candidate"], r["decision"])] = float(Decimal(r["net_proxy"])) +dates = sorted({d for c, d in v}) +M = np.array([[v[(c, d)] for c in trials] for d in dates]) +sd = M.std(axis=0) +sr = np.where(sd > 0, M.mean(axis=0) / np.where(sd > 0, sd, 1), 0.0) +b = int(np.argmax(sr)); x = M[:, b]; T = len(x) +d = x - x.mean(); g3 = (d**3).mean() / (d**2).mean()**1.5; g4 = (d**4).mean() / (d**2).mean()**2 +V = np.var(sr, ddof=1); N = len(trials); g = np.euler_gamma +sr0 = math.sqrt(V) * ((1 - g) * ppf(1 - 1 / N) + g * ppf(1 - 1 / (N * np.e))) +dsr = cdf((sr[b] - sr0) * math.sqrt(T - 1) / math.sqrt(1 - g3 * sr[b] + (g4 - 1) / 4 * sr[b] ** 2)) +# PBO, S=16, mean metric, direct slicing +S = 16; size = T // S; U = M[T - size * S:]; blocks = np.split(U, S) +lam = [] +for J in itertools.combinations(range(S), S // 2): + tr = np.vstack([blocks[i] for i in J]); te = np.vstack([blocks[i] for i in range(S) if i not in J]) + ism, osm = tr.mean(0), te.mean(0) + n = int(np.flatnonzero(ism == ism.max())[0]) + w = rankdata(osm)[n] / (len(trials) + 1) + lam.append(math.log(w / (1 - w))) +lam = np.array(lam) +print(json.dumps({"T": T, "selected": trials[b], "sr": sr[b], "sr0": sr0, "dsr": dsr, + "pbo": float((lam <= 0).mean()), "combinations": len(lam), + "numpy": np.__version__}, indent=1)) diff --git a/blueprints/us-equities/overfitting-controls/evaluate.py b/blueprints/us-equities/overfitting-controls/evaluate.py new file mode 100644 index 000000000..b1c782fa6 --- /dev/null +++ b/blueprints/us-equities/overfitting-controls/evaluate.py @@ -0,0 +1,165 @@ +#!/usr/bin/env python3 +"""Deflated Sharpe ratio and CSCV PBO over the retained research-evaluation ledger. + +Offline: reads only the hash-pinned ledger, the research-evaluation receipt and +params.json. Prints deterministic results JSON (no timestamps) to stdout. +""" +import argparse +from decimal import Decimal +import hashlib +import importlib.util +import json +import math +from pathlib import Path + +HERE = Path(__file__).resolve().parent +ROOT = HERE.parents[2] + + +def _library(): + spec = importlib.util.spec_from_file_location("overfitting_controls", HERE / "overfitting.py") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +of = _library() + + +def sha256(path): + return hashlib.sha256(Path(path).read_bytes()).hexdigest() + + +def pointer(document, path): + node = document + for token in path.strip("/").split("/"): + node = node[int(token)] if isinstance(node, list) else node[token] + return node + + +def load_inputs(root, params): + src = params["source"] + ledger_path, receipt_path = root / src["ledger_path"], root / src["research_receipt_path"] + if sha256(ledger_path) != src["ledger_sha256"]: + raise ValueError("ledger hash mismatch") + receipt = json.loads(receipt_path.read_text()) + if pointer(receipt, src["research_receipt_ledger_pointer"]) != src["ledger_sha256"]: + raise ValueError("research receipt does not name this ledger") + ledger = json.loads(ledger_path.read_text()) + if len(ledger) != src["ledger_records"]: + raise ValueError("ledger record count mismatch") + return ledger, receipt + + +def episode_values(ledger, development_end): + """One net_proxy per (candidate, decision) in the development segment; conflicts are errors.""" + values = {} + for row in ledger: + if row["status"] != "observed" or row["decision"] > development_end: + continue + key = (row["candidate"], row["decision"]) + value = Decimal(row["net_proxy"]) + if key in values and values[key] != value: + raise ValueError("conflicting duplicate episode label: %s %s" % key) + values[key] = value + return values + + +def aligned_matrix(values, trials): + dates = sorted({d for (c, d) in values if c in trials}) + missing = [(c, d) for c in trials for d in dates if (c, d) not in values] + if missing or not dates: + raise ValueError("trial series are not aligned") + return dates, [[float(values[(c, d)]) for c in trials] for d in dates] + + +def dsr_block(trials, matrix): + columns = list(zip(*matrix)) + per_trial = {} + for name, column in zip(trials, columns): + m = of.moments(column) + per_trial[name] = dict(m, sharpe=of.sharpe_from(m["mean"], m["std"])) + srs = [per_trial[t]["sharpe"] for t in trials] + best = max(range(len(trials)), key=lambda i: (srs[i], -i)) + selected = per_trial[trials[best]] + mean_sr = math.fsum(srs) / len(srs) + variance = math.fsum((s - mean_sr) ** 2 for s in srs) / (len(srs) - 1) + dsr, sr0 = of.deflated_sharpe(selected["sharpe"], selected["n"], selected["skew"], + selected["kurtosis"], len(trials), variance) + psr0 = of.probabilistic_sharpe(selected["sharpe"], selected["n"], selected["skew"], selected["kurtosis"], 0.0) + return {"trials": list(trials), "observations": len(matrix), "per_trial": per_trial, + "selected_trial": trials[best], "selected_sharpe": selected["sharpe"], + "trial_sharpe_variance_ddof1": variance, "expected_max_sharpe_sr0": sr0, + "psr_vs_zero": psr0, "dsr": dsr} + + +def pbo_block(matrix, partitions, metric): + result = of.cscv_pbo(matrix, partitions, metric) + logits = result.pop("logits") + histogram = {} + for x in logits: + key = "%.6f" % x + histogram[key] = histogram.get(key, 0) + 1 + result["lambda_histogram"] = dict(sorted(histogram.items(), key=lambda kv: float(kv[0]))) + return result + + +def walk_forward_block(ledger, receipt): + folds = receipt["results"]["development_folds"] + chosen = {f["id"]: f["chosen"] for f in folds} + series = sorted((r["decision"], float(Decimal(r["net_proxy"])), r["fold"]) + for r in ledger if r["phase"] == "evaluation" and r["status"] == "observed" + and r["fold"] in chosen and r["candidate"] == chosen[r["fold"]]) + decisions = [d for d, _, _ in series] + if len(set(decisions)) != len(decisions): + raise ValueError("overlapping walk-forward evaluation episodes") + values = [v for _, v, _ in series] + m = of.moments(values) + sr = of.sharpe_from(m["mean"], m["std"]) + return {"folds": len(chosen), "chosen_counts": {c: list(chosen.values()).count(c) for c in sorted(set(chosen.values()))}, + "observations": len(values), "first_decision": decisions[0], "last_decision": decisions[-1], + "moments": m, "sharpe": sr, "psr_vs_zero": of.probabilistic_sharpe(sr, m["n"], m["skew"], m["kurtosis"], 0.0)} + + +def compute(root=ROOT, params_path=HERE / "params.json"): + params = json.loads(Path(params_path).read_text()) + ledger, receipt = load_inputs(root, params) + values = episode_values(ledger, params["series"]["development_end"]) + dsr, pbo = {}, {} + grid = [params["pbo"]["partitions_primary"]] + params["pbo"]["partitions_sensitivity"] + metrics = [params["pbo"]["metric_primary"]] + params["pbo"]["metric_sensitivity"] + dates = None + for name, trials in params["trial_sets"].items(): + set_dates, matrix = aligned_matrix(values, trials) + if dates is None: + dates = set_dates + elif dates != set_dates: + raise ValueError("trial sets cover different decision dates") + dsr[name] = dsr_block(trials, matrix) + pbo[name] = {"%s_S%d" % (metric, s): pbo_block(matrix, s, metric) for metric in metrics for s in grid} + primary = pbo["primary"]["%s_S%d" % (params["pbo"]["metric_primary"], params["pbo"]["partitions_primary"])] + return { + "inputs": {"params_sha256": sha256(params_path), + "ledger_sha256": sha256(root / params["source"]["ledger_path"]), + "research_receipt_sha256": sha256(root / params["source"]["research_receipt_path"]), + "overfitting_py_sha256": sha256(HERE / "overfitting.py"), + "evaluate_py_sha256": sha256(HERE / "evaluate.py")}, + "ledger_records": len(ledger), + "development_episodes": len(dates), "first_decision": dates[0], "last_decision": dates[-1], + "dsr": dsr, "pbo": pbo, + "primary": {"dsr": dsr["primary"]["dsr"], "dsr_selected_trial": dsr["primary"]["selected_trial"], + "pbo": primary["pbo"], "pbo_strict": primary["pbo_strict"], + "probability_oos_loss": primary["probability_oos_loss"]}, + "walk_forward": walk_forward_block(ledger, receipt), + } + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--root", type=Path, default=ROOT) + args = parser.parse_args() + print(json.dumps(compute(args.root.resolve()), indent=2, sort_keys=True)) + + +if __name__ == "__main__": + main() diff --git a/blueprints/us-equities/overfitting-controls/overfitting.py b/blueprints/us-equities/overfitting-controls/overfitting.py new file mode 100644 index 000000000..d449ca176 --- /dev/null +++ b/blueprints/us-equities/overfitting-controls/overfitting.py @@ -0,0 +1,205 @@ +"""Backtest-overfitting controls written from the primary papers (standard library only). + +Sources (formulas transcribed from the papers; no third-party code was copied): + +* Probabilistic Sharpe ratio (PSR): D. H. Bailey and M. Lopez de Prado, + "The Sharpe Ratio Efficient Frontier", Journal of Risk 15(2), 2012, 3-44. + PSR(SR*) = Phi((SR - SR*) * sqrt(T - 1) / sqrt(1 - g3 * SR + (g4 - 1) / 4 * SR^2)), + with g3 the skewness and g4 the (non-excess) kurtosis of the returns. +* Deflated Sharpe ratio (DSR): D. H. Bailey and M. Lopez de Prado, "The Deflated + Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and + Non-Normality", Journal of Portfolio Management 40(5), 2014, 94-107 + (SSRN 2460551). DSR = PSR(SR0) with + SR0 = sqrt(V[SR_n]) * ((1 - gamma) * Phi^-1(1 - 1/N) + gamma * Phi^-1(1 - 1/(N e))), + gamma the Euler-Mascheroni constant, N the number of independent trials. +* Probability of backtest overfitting (PBO) by combinatorially symmetric + cross-validation (CSCV): D. H. Bailey, J. M. Borwein, M. Lopez de Prado and + Q. J. Zhu, "The Probability of Backtest Overfitting", Journal of + Computational Finance 20(4), 2017, 39-69 (SSRN 2326253). + +Sharpe ratios here are per-observation (not annualized) unless a caller scales them. +""" +from __future__ import annotations + +from itertools import combinations +import math +from statistics import NormalDist + +EULER_GAMMA = 0.5772156649015329 +_NORMAL = NormalDist() + + +def moments(values): + """Population mean, standard deviation, skewness and non-excess kurtosis.""" + xs = [float(x) for x in values] + n = len(xs) + if n < 2 or not all(math.isfinite(x) for x in xs): + raise ValueError("need at least two finite observations") + mean = math.fsum(xs) / n + dev = [x - mean for x in xs] + m2 = math.fsum(d * d for d in dev) / n + if m2 == 0.0: + return {"n": n, "mean": mean, "std": 0.0, "skew": None, "kurtosis": None} + m3 = math.fsum(d ** 3 for d in dev) / n + m4 = math.fsum(d ** 4 for d in dev) / n + return {"n": n, "mean": mean, "std": math.sqrt(m2), "skew": m3 / m2 ** 1.5, "kurtosis": m4 / m2 ** 2} + + +def sharpe_from(mean, std): + if std > 0.0: + return mean / std + if mean == 0.0: + return 0.0 + raise ValueError("zero-variance series with nonzero mean has no finite Sharpe ratio") + + +def sharpe(values): + stats = moments(values) + return sharpe_from(stats["mean"], stats["std"]) + + +def probabilistic_sharpe(sr, t, skew, kurtosis, sr_benchmark=0.0): + """PSR: probability that the true Sharpe exceeds sr_benchmark (Bailey and Lopez de Prado 2012).""" + if t < 2: + raise ValueError("need at least two observations") + variance_term = 1.0 - skew * sr + (kurtosis - 1.0) / 4.0 * sr * sr + if not variance_term > 0.0: + raise ValueError("non-positive Sharpe variance term") + z = (sr - sr_benchmark) * math.sqrt(t - 1) / math.sqrt(variance_term) + return _NORMAL.cdf(z) + + +def expected_max_sharpe(trials, variance): + """Expected maximum Sharpe of `trials` independent null trials (Bailey and Lopez de Prado 2014).""" + if not isinstance(trials, int) or trials < 1: + raise ValueError("trials must be a positive integer") + if not variance >= 0.0: + raise ValueError("variance must be nonnegative") + if trials == 1: + return 0.0 # no selection among trials: the null benchmark is zero + return math.sqrt(variance) * ((1.0 - EULER_GAMMA) * _NORMAL.inv_cdf(1.0 - 1.0 / trials) + + EULER_GAMMA * _NORMAL.inv_cdf(1.0 - 1.0 / (trials * math.e))) + + +def deflated_sharpe(sr, t, skew, kurtosis, trials, trial_variance): + """DSR = PSR evaluated at the expected maximum null Sharpe; returns (dsr, sr0).""" + sr0 = expected_max_sharpe(trials, trial_variance) + return probabilistic_sharpe(sr, t, skew, kurtosis, sr0), sr0 + + +def _merge(a, b): + """Chan et al. parallel combination of (count, mean, M2) aggregates.""" + n = a[0] + b[0] + delta = b[1] - a[1] + mean = a[1] + delta * b[0] / n + return (n, mean, a[2] + b[2] + delta * delta * a[0] * b[0] / n) + + +def _aggregate(rows): + n = len(rows) + mean = math.fsum(rows) / n + return (n, mean, math.fsum((x - mean) ** 2 for x in rows)) + + +def _score(aggregate, metric): + n, mean, m2 = aggregate + if metric == "mean": + return mean + if metric == "sharpe": + std = math.sqrt(max(m2, 0.0) / n) + if std <= 1e-12 * max(1.0, abs(mean)): + std = 0.0 # merged M2 of a constant column can carry rounding residue + return sharpe_from(mean, std) + raise ValueError("metric must be 'mean' or 'sharpe'") + + +def _average_ranks(values): + order = sorted(range(len(values)), key=lambda i: values[i]) + ranks = [0.0] * len(values) + i = 0 + while i < len(order): + j = i + while j + 1 < len(order) and values[order[j + 1]] == values[order[i]]: + j += 1 + for k in range(i, j + 1): + ranks[order[k]] = (i + j) / 2.0 + 1.0 + i = j + 1 + return ranks + + +def cscv_pbo(matrix, partitions=16, metric="mean"): + """Probability of backtest overfitting by CSCV. + + matrix: T rows (chronological observations) by N columns (trials). + The earliest T mod S rows are dropped so the S contiguous blocks are equal. + For every choice of S/2 blocks as the in-sample set J (complement J-bar out of + sample): n* = argmax in-sample metric (ties by column order), omega = average + ascending out-of-sample rank of n* / (N + 1), lambda = logit(omega). + PBO is the share of combinations with lambda <= 0. + + degradation_slope is the paper's pooled OLS slope of the selected trial's + out-of-sample metric on its in-sample metric. It is not evidence of ranking + persistence here. With metric="mean" and equal complementary halves, every + trial satisfies in_sample + out_of_sample = 2 * (its mean over the used rows), + so all splits that select the same trial lie on a line of slope exactly -1. + The pooled slope is therefore mostly this identity, mixed with the spread + between the full-sample means of the trials that get selected. With + metric="sharpe" the identity is not exact, but the complementary halves still + induce the same negative within-trial dependence. Judge persistence by PBO + against its no-skill level (about 0.5 for continuous ranks), not by the slope. + """ + rows = [[float(x) for x in row] for row in matrix] + if not rows or any(len(r) != len(rows[0]) for r in rows): + raise ValueError("matrix must be rectangular and nonempty") + t, n = len(rows), len(rows[0]) + if n < 2: + raise ValueError("need at least two trials") + if not isinstance(partitions, int) or partitions < 2 or partitions % 2: + raise ValueError("partitions must be an even integer >= 2") + size = t // partitions + if size < 2: + raise ValueError("each partition needs at least two rows") + if not all(math.isfinite(x) for r in rows for x in r): + raise ValueError("nonfinite performance value") + dropped = t - size * partitions + used = rows[dropped:] + blocks = [[_aggregate([used[b * size + i][c] for i in range(size)]) for c in range(n)] + for b in range(partitions)] + + def combine(indices, column): + agg = blocks[indices[0]][column] + for b in indices[1:]: + agg = _merge(agg, blocks[b][column]) + return agg + + logits, selected_is, selected_oos, chosen_counts = [], [], [], [0] * n + for chosen_blocks in combinations(range(partitions), partitions // 2): + rest = tuple(b for b in range(partitions) if b not in chosen_blocks) + in_sample = [_score(combine(chosen_blocks, c), metric) for c in range(n)] + out_sample = [_score(combine(rest, c), metric) for c in range(n)] + best = max(range(n), key=lambda c: (in_sample[c], -c)) + omega = _average_ranks(out_sample)[best] / (n + 1) + logits.append(math.log(omega / (1.0 - omega))) + selected_is.append(in_sample[best]) + selected_oos.append(out_sample[best]) + chosen_counts[best] += 1 + count = len(logits) + at_or_below = sum(x <= 0.0 for x in logits) + zero = sum(x == 0.0 for x in logits) + mean_is = math.fsum(selected_is) / count + mean_oos = math.fsum(selected_oos) / count + sxx = math.fsum((x - mean_is) ** 2 for x in selected_is) + slope = (math.fsum((x - mean_is) * (y - mean_oos) for x, y in zip(selected_is, selected_oos)) / sxx + if sxx > 0 else None) + return { + "partitions": partitions, "metric": metric, "rows": t, "rows_used": len(used), + "rows_dropped_earliest": dropped, "rows_per_partition": size, "trials": n, + "combinations": count, "pbo": at_or_below / count, "pbo_strict": (at_or_below - zero) / count, + "lambda_zero_count": zero, "lambda_mean": math.fsum(logits) / count, + "lambda_min": min(logits), "lambda_max": max(logits), + "probability_oos_loss": sum(y < 0.0 for y in selected_oos) / count, + "selected_in_sample_mean": mean_is, "selected_out_of_sample_mean": mean_oos, + "degradation_slope": slope, + "degradation_intercept": (mean_oos - slope * mean_is) if slope is not None else None, + "in_sample_selection_counts": chosen_counts, "logits": logits, + } diff --git a/blueprints/us-equities/overfitting-controls/params.json b/blueprints/us-equities/overfitting-controls/params.json new file mode 100644 index 000000000..7618d4c6b --- /dev/null +++ b/blueprints/us-equities/overfitting-controls/params.json @@ -0,0 +1,70 @@ +{ + "schema_version": 1, + "id": "overfitting-controls-research-evaluation-v1", + "frozen_before_computation": true, + "purpose": "Deflated Sharpe ratio and CSCV probability of backtest overfitting over the retained multi-candidate research-evaluation ledger. Diagnostic only; no selection, tuning or promotion follows from the result.", + "source": { + "ledger_path": "evidence/artifacts/gap-wave2-20260923/us-equities__portfolio-risk/raw/3/regenerated/run-1/candidate-ledger.json", + "ledger_sha256": "08ea5309466045574ba4b9c8524608ed66c91edb22cb592a5bad84e1e2f42672", + "ledger_records": 6605, + "research_receipt_path": "blueprints/us-equities/research-evaluation/receipt.json", + "research_receipt_ledger_pointer": "/results/ledger_sha256", + "provenance": "Byte-identical regeneration of the private run-1 ledger whose SHA-256 the research-evaluation receipt records; the regeneration comparison is the gap-wave2 portfolio-risk raw/3 compare.json." + }, + "series": { + "development_end": "2020-12-31", + "segment": "development only: observed labels whose decision date is on or before 2020-12-31; reserved 2021Q1 labels are excluded", + "value": "net_proxy (five-session open-to-open raw-price label less the fixed 20bp cost proxy), parsed from its decimal string", + "deduplication": "One value per (candidate, decision). Training and evaluation copies of the same episode must be identical; any conflict is an error.", + "alignment": "Every trial must have an observed value at every retained decision date; otherwise an error." + }, + "trial_sets": { + "primary": [ + "momentum20", + "momentum60", + "momentum120", + "cash" + ], + "with_equalweight_control": [ + "momentum20", + "momentum60", + "momentum120", + "cash", + "equalweight" + ], + "momentum_only": [ + "momentum20", + "momentum60", + "momentum120" + ] + }, + "dsr": { + "sharpe": "Per-episode (five-session) mean divided by population standard deviation; not annualized. A zero-variance series with zero mean (cash) has Sharpe 0.", + "moments": "Population skewness and non-excess kurtosis of the selected trial's series (Bailey and Lopez de Prado 2014 gamma3, gamma4).", + "selected_trial": "Highest full-development Sharpe within the trial set; ties by trial-set order.", + "trial_count": "Number of trials in the set (no effective-N clustering).", + "trial_variance": "Sample variance (ddof=1) of the trial Sharpe estimates.", + "benchmark": "Expected maximum Sharpe under the null, SR0 = sqrt(V) * ((1-gamma) * Phi^-1(1-1/N) + gamma * Phi^-1(1-1/(N e)))." + }, + "pbo": { + "method": "CSCV (Bailey, Borwein, Lopez de Prado and Zhu 2017)", + "partitions_primary": 16, + "partitions_sensitivity": [ + 8, + 10, + 12 + ], + "metric_primary": "mean", + "metric_sensitivity": [ + "sharpe" + ], + "truncation": "Drop the earliest T mod S rows so that S contiguous blocks have equal size.", + "in_sample_tie_break": "Trial-set order (the frozen plan's tie rule).", + "out_of_sample_rank": "Ascending rank among the N trials with average ranks for ties; omega = rank / (N + 1); lambda = ln(omega / (1 - omega)).", + "pbo_rule": "Share of combinations with lambda <= 0 (primary). The strict lambda < 0 share and the count of lambda == 0 are reported alongside." + }, + "walk_forward": { + "series": "Development evaluation labels of each development fold's frozen chosen candidate (from the research-evaluation receipt), observed only.", + "statistic": "Probabilistic Sharpe ratio against a zero benchmark; the walk-forward selector is one pre-registered procedure, so no additional trial deflation is applied to it." + } +} diff --git a/blueprints/us-equities/overfitting-controls/receipt.json b/blueprints/us-equities/overfitting-controls/receipt.json new file mode 100644 index 000000000..405845a46 --- /dev/null +++ b/blueprints/us-equities/overfitting-controls/receipt.json @@ -0,0 +1,1153 @@ +{ + "schema_version": 1, + "id": "overfitting-controls-research-evaluation-20260924", + "evidence_class": "local_integration_check", + "observed_utc": "2026-09-24T15:54:19Z", + "status": "diagnostic_computed_no_promotion", + "claim": "Deflated Sharpe ratio (Bailey and Lopez de Prado 2014) and CSCV probability of backtest overfitting (Bailey, Borwein, Lopez de Prado and Zhu 2017) were computed offline over the 6605-record research-evaluation candidate ledger (301 aligned development episodes, 2015-01-02 to 2020-12-16). For the frozen four-candidate selection set the highest-Sharpe candidate (momentum120, per-episode Sharpe 0.0254) has DSR 0.5521 against SR0 0.0177, and PBO is 0.7756 (S=16, mean-label metric, 12870 combinations). The walk-forward selected series has Sharpe -0.0151 and PSR(0) 0.4048. The retained candidates show no deflated-Sharpe evidence of skill, and the in-sample best candidate usually ranks below the out-of-sample median.", + "frozen_parameters": { + "path": "blueprints/us-equities/overfitting-controls/params.json", + "sha256": "0e371da46601c673453a40ebcb608e38512348c37bf95a63ac7793d263c21487", + "frozen_utc": "2026-09-24T15:53:14Z", + "before": "The first computation over the retained ledger (run-1 started 2026-09-24T15:54:19Z). Library unit tests on synthetic fixtures and the published example ran before it; no parameter changed after results were seen." + }, + "inputs": { + "ledger": { + "path": "evidence/artifacts/gap-wave2-20260923/us-equities__portfolio-risk/raw/3/regenerated/run-1/candidate-ledger.json", + "sha256": "08ea5309466045574ba4b9c8524608ed66c91edb22cb592a5bad84e1e2f42672", + "bytes": 2536508, + "records": 6605, + "provenance": "SHA-256 equals results.ledger_sha256 in blueprints/us-equities/research-evaluation/receipt.json (native skfolio 1.2.9 run of 2026-09-19); this in-tree copy is the gap-wave2 portfolio-risk byte-identical regeneration (raw/3/compare.json). No data was refetched." + }, + "research_receipt": { + "path": "blueprints/us-equities/research-evaluation/receipt.json", + "sha256": "92a2c4c1984c7cd38af21fbc9b5290395805fed3177522b19d408ad50c59d56b", + "used_for": "Ledger hash binding and each development fold's frozen chosen candidate for the walk-forward series." + } + }, + "method": { + "primary_sources": [ + { + "citation": "Bailey, D. H. and Lopez de Prado, M. (2012). The Sharpe Ratio Efficient Frontier. Journal of Risk 15(2), 3-44.", + "used_for": "Probabilistic Sharpe ratio with skewness and kurtosis adjustment." + }, + { + "citation": "Bailey, D. H. and Lopez de Prado, M. (2014). The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-Normality. Journal of Portfolio Management 40(5), 94-107. SSRN 2460551.", + "used_for": "Expected maximum null Sharpe SR0 and DSR = PSR(SR0); the paper's numerical example is a unit test." + }, + { + "citation": "Bailey, D. H., Borwein, J. M., Lopez de Prado, M. and Zhu, Q. J. (2017). The Probability of Backtest Overfitting. Journal of Computational Finance 20(4), 39-69. SSRN 2326253.", + "used_for": "CSCV algorithm, relative rank logits, PBO, probability of out-of-sample loss and performance degradation regression." + } + ], + "implementation_decision": { + "choice": "Written from the papers as a standard-library module (overfitting.py); no third-party code copied or vendored.", + "alternatives_considered": [ + "eslazarev/purged-cross-validation: catalogued discovery_only in catalogs/landscape/claude-independent-discovery.json with no source pin, licence review or execution in this repository.", + "pypbo (Python) and the R pbo package: no copy on this host and no reviewed pin; installing either would require a network fetch and a new dependency lock for roughly 150 lines of arithmetic.", + "skfolio 1.2.9 (already locked for this lane) supplied the WalkForward splitter upstream of this ledger; this change did not review skfolio for DSR or PBO estimators." + ], + "overturn_condition": "Prefer a maintained, licence-reviewed upstream implementation if, pinned and executed, it reproduces the DSR numerical example and these ledger results within floating-point tolerance; also revisit if the papers' text contradicts a transcribed formula." + }, + "parameters": "See frozen params.json: trial sets, per-episode (non-annualized) Sharpe with population moments, ddof=1 trial variance, S=16 primary with 8/10/12 sensitivity, mean-label primary metric with Sharpe sensitivity, earliest-row truncation, trial-order in-sample ties, average out-of-sample ranks, PBO rule lambda <= 0." + }, + "implementation": { + "overfitting.py": { + "path": "blueprints/us-equities/overfitting-controls/overfitting.py", + "sha256": "1747e058c0a5830db4ae43f4101cdd4a6825422f4b5b386fd8eb7e51f803811a" + }, + "evaluate.py": { + "path": "blueprints/us-equities/overfitting-controls/evaluate.py", + "sha256": "e62af085613c0b341ffdfc2a0eac94255551325f430844f29e317a26ede6cedc" + }, + "crosscheck.py": { + "path": "blueprints/us-equities/overfitting-controls/crosscheck.py", + "sha256": "dc03b73d54ecf64f4286e98cab7242898979a05247c2042729efe80b663d03d0" + } + }, + "commands": [ + { + "id": "run-1", + "argv": [ + "python3", + "blueprints/us-equities/overfitting-controls/evaluate.py" + ], + "cwd": "checkout root", + "runtime": "CPython 3.13.15 (standard library only)", + "started_utc": "2026-09-24T15:54:19Z", + "ended_utc": "2026-09-24T15:54:21Z", + "exit": 0, + "stderr": "only the elapsed-time line of the wrapping /usr/bin/time (1.54 s)", + "stdout_sha256": "3815782f85fcb7c8aebd6defce82ae9540039f6bf408a82a83ff47dd0a83ff43" + }, + { + "id": "run-2", + "argv": [ + "python3", + "blueprints/us-equities/overfitting-controls/evaluate.py" + ], + "runtime": "CPython 3.13.15", + "exit": 0, + "stderr_bytes": 0, + "stdout_sha256": "3815782f85fcb7c8aebd6defce82ae9540039f6bf408a82a83ff47dd0a83ff43" + }, + { + "id": "run-3", + "argv": [ + "", + "blueprints/us-equities/overfitting-controls/evaluate.py" + ], + "runtime": "CPython 3.12.3", + "exit": 0, + "stderr_bytes": 0, + "stdout_sha256": "3815782f85fcb7c8aebd6defce82ae9540039f6bf408a82a83ff47dd0a83ff43" + }, + { + "id": "fix-round-runs", + "argv": [ + "python3", + "blueprints/us-equities/overfitting-controls/evaluate.py" + ], + "cwd": "checkout root", + "runtime": "CPython 3.13.15 (two runs) and CPython 3.12.3 pinned adaptive-paper runtime (one run)", + "started_utc": "2026-09-24T16:22:00Z", + "exit": 0, + "stderr_bytes": 0, + "stdout_sha256": "6bacc72f585348e624a63826ce1b65d30773fd8c50fe02e7357ebf9ceea6dbe5", + "note": "Re-run after the review fix that documented the degradation-slope identity in the overfitting.py docstring. All three stdouts are byte-identical. Only results.inputs.overfitting_py_sha256 differs from runs 1-3; every numeric result is unchanged. Runs 1-3 are kept as the original observation; their stdout hash predates the docstring change." + } + ], + "determinism": "Runs 1-3 produced byte-identical stdout (3815782f...). After the docstring-only fix, three further runs (fix-round-runs) produced byte-identical stdout (6bacc72f...), differing from the original only in the overfitting.py source hash. The results object below is the fix-round stdout.", + "independent_observation": { + "method": "crosscheck.py recomputes the primary DSR and PBO with numpy 2.5.3 (array moments, direct row slicing of all 12870 CSCV combinations, explicit average ranks, math.erf CDF), sharing only statistics.NormalDist.inv_cdf and the ledger with the module under test.", + "runtime": "CPython 3.12.3 pinned adaptive-paper runtime, exit 0, empty stderr", + "stdout_sha256": "16fa24af60088749727bea5f789136a44d03be78e68dd018ce6a0db788a63395", + "result": { + "T": 301, + "selected": "momentum120", + "sr": 0.02539620742431157, + "sr0": 0.017690872785666837, + "dsr": 0.5520685371554236, + "pbo": 0.7756021756021756, + "combinations": 12870, + "numpy": "2.5.3" + }, + "matches_primary": true + }, + "tests": { + "file": "tests/test_overfitting_controls.py", + "command": "python3 -m unittest tests.test_overfitting_controls -v", + "classes": { + "published_worked_example": "Bailey and Lopez de Prado (2014) numerical example (N=100, annualized V=1/2, SR=2.5, T=1250, skew -3, kurtosis 10): SR0 0.1132 and DSR 0.9004 reproduced to four decimals. Expected values are transcribed from the paper; the paper text was not re-fetched in this offline session.", + "reference_value": "Expected maximum of 100 iid standard normals (order-statistic tables, 2.50759) within the approximation's 1.5% tolerance.", + "synthetic_fixture": "Locally constructed PBO cases: exact reversal (PBO 1), dominant trial (PBO 0), pure-noise mean near 0.5 over 20 seeds, block-merge equality with a brute-force recomputation, tie handling, the mean-metric degradation-slope identity (slope exactly -1 for an always-selected trial; the selected in-sample plus out-of-sample means equal twice the selection-weighted full means) and invalid configurations. No published PBO numerical example with exact expected values was available offline.", + "replay": "Recomputes this receipt's results object from the retained ledger (float tolerance 1e-9 relative)." + }, + "observed": { + "focused": "After the fix round: 14 tests passed, 0 skipped on CPython 3.13.15; with tests.test_research_evaluation, 26 passed, 0 skipped on CPython 3.12.3 (pinned runtime)", + "full_suite": "python3 -m unittest on CPython 3.13.15 (system interpreter without numpy, jsonschema or PyYAML): 4032 tests OK, 490 skipped; none of the skips are in tests.test_overfitting_controls" + } + }, + "results": { + "development_episodes": 301, + "dsr": { + "momentum_only": { + "dsr": 0.5608136084061657, + "expected_max_sharpe_sr0": 0.016387328850452767, + "observations": 301, + "per_trial": { + "momentum120": { + "kurtosis": 11.70581994534825, + "mean": 0.000647213217448946, + "n": 301, + "sharpe": 0.02539620742431157, + "skew": -1.4941526859828613, + "std": 0.025484640546342932 + }, + "momentum20": { + "kurtosis": 10.534059391680314, + "mean": 0.00047619722098306353, + "n": 301, + "sharpe": 0.020820777225615553, + "skew": -0.6864816724012911, + "std": 0.022871250953936714 + }, + "momentum60": { + "kurtosis": 14.863770264503078, + "mean": -0.00022989605827723083, + "n": 301, + "sharpe": -0.00993755279683209, + "skew": -2.2018993871687806, + "std": 0.02313407163487146 + } + }, + "psr_vs_zero": 0.6669116476097554, + "selected_sharpe": 0.02539620742431157, + "selected_trial": "momentum120", + "trial_sharpe_variance_ddof1": 0.0003692473397730362, + "trials": [ + "momentum20", + "momentum60", + "momentum120" + ] + }, + "primary": { + "dsr": 0.5520685371554236, + "expected_max_sharpe_sr0": 0.01769087278566683, + "observations": 301, + "per_trial": { + "cash": { + "kurtosis": null, + "mean": 0.0, + "n": 301, + "sharpe": 0.0, + "skew": null, + "std": 0.0 + }, + "momentum120": { + "kurtosis": 11.70581994534825, + "mean": 0.000647213217448946, + "n": 301, + "sharpe": 0.02539620742431157, + "skew": -1.4941526859828613, + "std": 0.025484640546342932 + }, + "momentum20": { + "kurtosis": 10.534059391680314, + "mean": 0.00047619722098306353, + "n": 301, + "sharpe": 0.020820777225615553, + "skew": -0.6864816724012911, + "std": 0.022871250953936714 + }, + "momentum60": { + "kurtosis": 14.863770264503078, + "mean": -0.00022989605827723083, + "n": 301, + "sharpe": -0.00993755279683209, + "skew": -2.2018993871687806, + "std": 0.02313407163487146 + } + }, + "psr_vs_zero": 0.6669116476097554, + "selected_sharpe": 0.02539620742431157, + "selected_trial": "momentum120", + "trial_sharpe_variance_ddof1": 0.0002827259258371177, + "trials": [ + "momentum20", + "momentum60", + "momentum120", + "cash" + ] + }, + "with_equalweight_control": { + "dsr": 0.5750830794792382, + "expected_max_sharpe_sr0": 0.02141737891040903, + "observations": 301, + "per_trial": { + "cash": { + "kurtosis": null, + "mean": 0.0, + "n": 301, + "sharpe": 0.0, + "skew": null, + "std": 0.0 + }, + "equalweight": { + "kurtosis": 11.325886200438843, + "mean": 0.0008972410662137079, + "n": 301, + "sharpe": 0.03257184566045634, + "skew": -1.1840401587874543, + "std": 0.02754652209662771 + }, + "momentum120": { + "kurtosis": 11.70581994534825, + "mean": 0.000647213217448946, + "n": 301, + "sharpe": 0.02539620742431157, + "skew": -1.4941526859828613, + "std": 0.025484640546342932 + }, + "momentum20": { + "kurtosis": 10.534059391680314, + "mean": 0.00047619722098306353, + "n": 301, + "sharpe": 0.020820777225615553, + "skew": -0.6864816724012911, + "std": 0.022871250953936714 + }, + "momentum60": { + "kurtosis": 14.863770264503078, + "mean": -0.00022989605827723083, + "n": 301, + "sharpe": -0.00993755279683209, + "skew": -2.2018993871687806, + "std": 0.02313407163487146 + } + }, + "psr_vs_zero": 0.7098198477046372, + "selected_sharpe": 0.03257184566045634, + "selected_trial": "equalweight", + "trial_sharpe_variance_ddof1": 0.00032251312952154255, + "trials": [ + "momentum20", + "momentum60", + "momentum120", + "cash", + "equalweight" + ] + } + }, + "first_decision": "2015-01-02", + "inputs": { + "evaluate_py_sha256": "e62af085613c0b341ffdfc2a0eac94255551325f430844f29e317a26ede6cedc", + "ledger_sha256": "08ea5309466045574ba4b9c8524608ed66c91edb22cb592a5bad84e1e2f42672", + "overfitting_py_sha256": "1747e058c0a5830db4ae43f4101cdd4a6825422f4b5b386fd8eb7e51f803811a", + "params_sha256": "0e371da46601c673453a40ebcb608e38512348c37bf95a63ac7793d263c21487", + "research_receipt_sha256": "92a2c4c1984c7cd38af21fbc9b5290395805fed3177522b19d408ad50c59d56b" + }, + "last_decision": "2020-12-16", + "ledger_records": 6605, + "pbo": { + "momentum_only": { + "mean_S10": { + "combinations": 252, + "degradation_intercept": 0.0008984119858695838, + "degradation_slope": -0.9118725676377238, + "in_sample_selection_counts": [ + 107, + 25, + 120 + ], + "lambda_histogram": { + "-1.098612": 116, + "0.000000": 120, + "1.098612": 16 + }, + "lambda_max": 1.0986122886681098, + "lambda_mean": -0.4359572574079801, + "lambda_min": -1.0986122886681098, + "lambda_zero_count": 120, + "metric": "mean", + "partitions": 10, + "pbo": 0.9365079365079365, + "pbo_strict": 0.4603174603174603, + "probability_oos_loss": 0.5317460317460317, + "rows": 301, + "rows_dropped_earliest": 1, + "rows_per_partition": 30, + "rows_used": 300, + "selected_in_sample_mean": 0.0011109767822259436, + "selected_out_of_sample_mean": -0.00011465726512468383, + "trials": 3 + }, + "mean_S12": { + "combinations": 924, + "degradation_intercept": 0.0009573417091899579, + "degradation_slope": -0.9115018430994034, + "in_sample_selection_counts": [ + 408, + 61, + 455 + ], + "lambda_histogram": { + "-1.098612": 343, + "0.000000": 511, + "1.098612": 70 + }, + "lambda_max": 1.0986122886681098, + "lambda_mean": -0.32458999437921426, + "lambda_min": -1.0986122886681098, + "lambda_zero_count": 511, + "metric": "mean", + "partitions": 12, + "pbo": 0.9242424242424242, + "pbo_strict": 0.3712121212121212, + "probability_oos_loss": 0.48593073593073594, + "rows": 301, + "rows_dropped_earliest": 1, + "rows_per_partition": 25, + "rows_used": 300, + "selected_in_sample_mean": 0.001025869501173754, + "selected_out_of_sample_mean": 2.2259768090615506e-05, + "trials": 3 + }, + "mean_S16": { + "combinations": 12870, + "degradation_intercept": 0.0008912723722196319, + "degradation_slope": -0.9211665387182658, + "in_sample_selection_counts": [ + 5746, + 776, + 6348 + ], + "lambda_histogram": { + "-1.098612": 4428, + "0.000000": 7880, + "1.098612": 562 + }, + "lambda_max": 1.0986122886681098, + "lambda_mean": -0.33001049790139175, + "lambda_min": -1.0986122886681098, + "lambda_zero_count": 7880, + "metric": "mean", + "partitions": 16, + "pbo": 0.9563325563325563, + "pbo_strict": 0.34405594405594403, + "probability_oos_loss": 0.4966588966588967, + "rows": 301, + "rows_dropped_earliest": 13, + "rows_per_partition": 18, + "rows_used": 288, + "selected_in_sample_mean": 0.0009689287943105048, + "selected_out_of_sample_mean": -1.2724114998384463e-06, + "trials": 3 + }, + "mean_S8": { + "combinations": 70, + "degradation_intercept": 0.0009605227808458237, + "degradation_slope": -0.9140463245423784, + "in_sample_selection_counts": [ + 29, + 7, + 34 + ], + "lambda_histogram": { + "-1.098612": 25, + "0.000000": 39, + "1.098612": 6 + }, + "lambda_max": 1.0986122886681098, + "lambda_mean": -0.29819476406705836, + "lambda_min": -1.0986122886681098, + "lambda_zero_count": 39, + "metric": "mean", + "partitions": 8, + "pbo": 0.9142857142857143, + "pbo_strict": 0.35714285714285715, + "probability_oos_loss": 0.5428571428571428, + "rows": 301, + "rows_dropped_earliest": 5, + "rows_per_partition": 37, + "rows_used": 296, + "selected_in_sample_mean": 0.0009350556555339403, + "selected_out_of_sample_mean": 0.00010583859566246121, + "trials": 3 + }, + "sharpe_S10": { + "combinations": 252, + "degradation_intercept": 0.02990585230917518, + "degradation_slope": -0.9276892193206676, + "in_sample_selection_counts": [ + 101, + 27, + 124 + ], + "lambda_histogram": { + "-1.098612": 120, + "0.000000": 108, + "1.098612": 24 + }, + "lambda_max": 1.0986122886681098, + "lambda_mean": -0.41851896711166087, + "lambda_min": -1.0986122886681098, + "lambda_zero_count": 108, + "metric": "sharpe", + "partitions": 10, + "pbo": 0.9047619047619048, + "pbo_strict": 0.47619047619047616, + "probability_oos_loss": 0.5357142857142857, + "rows": 301, + "rows_dropped_earliest": 1, + "rows_per_partition": 30, + "rows_used": 300, + "selected_in_sample_mean": 0.04271119999452696, + "selected_out_of_sample_mean": -0.009716867469996438, + "trials": 3 + }, + "sharpe_S12": { + "combinations": 924, + "degradation_intercept": 0.03473384022372467, + "degradation_slope": -0.9003237991945331, + "in_sample_selection_counts": [ + 395, + 72, + 457 + ], + "lambda_histogram": { + "-1.098612": 353, + "0.000000": 477, + "1.098612": 94 + }, + "lambda_max": 1.0986122886681098, + "lambda_mean": -0.30794435364181866, + "lambda_min": -1.0986122886681098, + "lambda_zero_count": 477, + "metric": "sharpe", + "partitions": 12, + "pbo": 0.8982683982683982, + "pbo_strict": 0.38203463203463206, + "probability_oos_loss": 0.4945887445887446, + "rows": 301, + "rows_dropped_earliest": 1, + "rows_per_partition": 25, + "rows_used": 300, + "selected_in_sample_mean": 0.040848690296514785, + "selected_out_of_sample_mean": -0.0020432078161543806, + "trials": 3 + }, + "sharpe_S16": { + "combinations": 12870, + "degradation_intercept": 0.028851212592936295, + "degradation_slope": -0.8725597711548888, + "in_sample_selection_counts": [ + 5373, + 990, + 6507 + ], + "lambda_histogram": { + "-1.098612": 4800, + "0.000000": 7032, + "1.098612": 1038 + }, + "lambda_max": 1.0986122886681098, + "lambda_mean": -0.32113282284144745, + "lambda_min": -1.0986122886681098, + "lambda_zero_count": 7032, + "metric": "sharpe", + "partitions": 16, + "pbo": 0.9193473193473194, + "pbo_strict": 0.372960372960373, + "probability_oos_loss": 0.498989898989899, + "rows": 301, + "rows_dropped_earliest": 13, + "rows_per_partition": 18, + "rows_used": 288, + "selected_in_sample_mean": 0.03685917292888655, + "selected_out_of_sample_mean": -0.0033106189028514247, + "trials": 3 + }, + "sharpe_S8": { + "combinations": 70, + "degradation_intercept": 0.030083454037995377, + "degradation_slope": -0.9262724087164941, + "in_sample_selection_counts": [ + 23, + 8, + 39 + ], + "lambda_histogram": { + "-1.098612": 26, + "0.000000": 28, + "1.098612": 16 + }, + "lambda_max": 1.0986122886681098, + "lambda_mean": -0.15694461266687285, + "lambda_min": -1.0986122886681098, + "lambda_zero_count": 28, + "metric": "sharpe", + "partitions": 8, + "pbo": 0.7714285714285715, + "pbo_strict": 0.37142857142857144, + "probability_oos_loss": 0.5428571428571428, + "rows": 301, + "rows_dropped_earliest": 5, + "rows_per_partition": 37, + "rows_used": 296, + "selected_in_sample_mean": 0.03183331097631105, + "selected_out_of_sample_mean": 0.0005971364025465317, + "trials": 3 + } + }, + "primary": { + "mean_S10": { + "combinations": 252, + "degradation_intercept": 0.0003755960182084304, + "degradation_slope": -0.6974169642087382, + "in_sample_selection_counts": [ + 73, + 12, + 98, + 69 + ], + "lambda_histogram": { + "-0.405465": 88, + "-1.386294": 125, + "0.405465": 33, + "1.386294": 6 + }, + "lambda_max": 1.3862943611198908, + "lambda_mean": -0.7431333726953017, + "lambda_min": -1.3862943611198906, + "lambda_zero_count": 0, + "metric": "mean", + "partitions": 10, + "pbo": 0.8452380952380952, + "pbo_strict": 0.8452380952380952, + "probability_oos_loss": 0.5119047619047619, + "rows": 301, + "rows_dropped_earliest": 1, + "rows_per_partition": 30, + "rows_used": 300, + "selected_in_sample_mean": 0.0013061708589959147, + "selected_out_of_sample_mean": -0.0005353496970104204, + "trials": 4 + }, + "mean_S12": { + "combinations": 924, + "degradation_intercept": 0.0003975356503207397, + "degradation_slope": -0.6748833832155265, + "in_sample_selection_counts": [ + 255, + 31, + 386, + 252 + ], + "lambda_histogram": { + "-0.405465": 271, + "-1.386294": 449, + "0.405465": 182, + "1.386294": 22 + }, + "lambda_max": 1.3862943611198908, + "lambda_mean": -0.6796905701513202, + "lambda_min": -1.3862943611198906, + "lambda_zero_count": 0, + "metric": "mean", + "partitions": 12, + "pbo": 0.7792207792207793, + "pbo_strict": 0.7792207792207793, + "probability_oos_loss": 0.4816017316017316, + "rows": 301, + "rows_dropped_earliest": 1, + "rows_per_partition": 25, + "rows_used": 300, + "selected_in_sample_mean": 0.0012389532857593503, + "selected_out_of_sample_mean": -0.0004386133348185236, + "trials": 4 + }, + "mean_S16": { + "combinations": 12870, + "degradation_intercept": 0.00044423801083574936, + "degradation_slope": -0.7273275972514565, + "in_sample_selection_counts": [ + 3814, + 289, + 5490, + 3277 + ], + "lambda_histogram": { + "-0.405465": 4128, + "-1.386294": 5854, + "0.405465": 2632, + "1.386294": 256 + }, + "lambda_max": 1.3862943611198908, + "lambda_mean": -0.6501205621817375, + "lambda_min": -1.3862943611198906, + "lambda_zero_count": 0, + "metric": "mean", + "partitions": 16, + "pbo": 0.7756021756021756, + "pbo_strict": 0.7756021756021756, + "probability_oos_loss": 0.48003108003108, + "rows": 301, + "rows_dropped_earliest": 13, + "rows_per_partition": 18, + "rows_used": 288, + "selected_in_sample_mean": 0.0011880286000091778, + "selected_out_of_sample_mean": -0.00041984797627493757, + "trials": 4 + }, + "mean_S8": { + "combinations": 70, + "degradation_intercept": 0.00012216926410789342, + "degradation_slope": -0.6075829596038457, + "in_sample_selection_counts": [ + 12, + 2, + 26, + 30 + ], + "lambda_histogram": { + "-0.405465": 18, + "-1.386294": 45, + "0.405465": 7 + }, + "lambda_max": 0.4054651081081642, + "lambda_mean": -0.9549051777083554, + "lambda_min": -1.3862943611198906, + "lambda_zero_count": 0, + "metric": "mean", + "partitions": 8, + "pbo": 0.9, + "pbo_strict": 0.9, + "probability_oos_loss": 0.5142857142857142, + "rows": 301, + "rows_dropped_earliest": 5, + "rows_per_partition": 37, + "rows_used": 296, + "selected_in_sample_mean": 0.0013772289987410554, + "selected_out_of_sample_mean": -0.0007146116069994381, + "trials": 4 + }, + "sharpe_S10": { + "combinations": 252, + "degradation_intercept": 0.014650477575139365, + "degradation_slope": -0.7779968069155431, + "in_sample_selection_counts": [ + 73, + 14, + 96, + 69 + ], + "lambda_histogram": { + "-0.405465": 86, + "-1.386294": 125, + "0.405465": 37, + "1.386294": 4 + }, + "lambda_max": 1.3862943611198908, + "lambda_mean": -0.7444817777492334, + "lambda_min": -1.3862943611198906, + "lambda_zero_count": 0, + "metric": "sharpe", + "partitions": 10, + "pbo": 0.8373015873015873, + "pbo_strict": 0.8373015873015873, + "probability_oos_loss": 0.5158730158730159, + "rows": 301, + "rows_dropped_earliest": 1, + "rows_per_partition": 30, + "rows_used": 300, + "selected_in_sample_mean": 0.052143438043757, + "selected_out_of_sample_mean": -0.025916950724502032, + "trials": 4 + }, + "sharpe_S12": { + "combinations": 924, + "degradation_intercept": 0.013853123632035766, + "degradation_slope": -0.6874544874643813, + "in_sample_selection_counts": [ + 269, + 40, + 363, + 252 + ], + "lambda_histogram": { + "-0.405465": 266, + "-1.386294": 446, + "0.405465": 200, + "1.386294": 12 + }, + "lambda_max": 1.3862943611198908, + "lambda_mean": -0.6801000539623067, + "lambda_min": -1.3862943611198906, + "lambda_zero_count": 0, + "metric": "sharpe", + "partitions": 12, + "pbo": 0.7705627705627706, + "pbo_strict": 0.7705627705627706, + "probability_oos_loss": 0.4902597402597403, + "rows": 301, + "rows_dropped_earliest": 1, + "rows_per_partition": 25, + "rows_used": 300, + "selected_in_sample_mean": 0.05072668963403052, + "selected_out_of_sample_mean": -0.021019166791091427, + "trials": 4 + }, + "sharpe_S16": { + "combinations": 12870, + "degradation_intercept": 0.01674539901662133, + "degradation_slope": -0.7513857960806316, + "in_sample_selection_counts": [ + 3964, + 336, + 5293, + 3277 + ], + "lambda_histogram": { + "-0.405465": 3960, + "-1.386294": 5770, + "0.405465": 3056, + "1.386294": 84 + }, + "lambda_max": 1.3862943611198908, + "lambda_mean": -0.6409487331046992, + "lambda_min": -1.3862943611198906, + "lambda_zero_count": 0, + "metric": "sharpe", + "partitions": 16, + "pbo": 0.756021756021756, + "pbo_strict": 0.756021756021756, + "probability_oos_loss": 0.48166278166278165, + "rows": 301, + "rows_dropped_earliest": 13, + "rows_per_partition": 18, + "rows_used": 288, + "selected_in_sample_mean": 0.047414314217345, + "selected_out_of_sample_mean": -0.01888104321719565, + "trials": 4 + }, + "sharpe_S8": { + "combinations": 70, + "degradation_intercept": 0.0027391817604871457, + "degradation_slope": -0.6594005137910001, + "in_sample_selection_counts": [ + 12, + 2, + 26, + 30 + ], + "lambda_histogram": { + "-0.405465": 15, + "-1.386294": 43, + "0.405465": 12 + }, + "lambda_max": 0.4054651081081642, + "lambda_mean": -0.8689578978925684, + "lambda_min": -1.3862943611198906, + "lambda_zero_count": 0, + "metric": "sharpe", + "partitions": 8, + "pbo": 0.8285714285714286, + "pbo_strict": 0.8285714285714286, + "probability_oos_loss": 0.5142857142857142, + "rows": 301, + "rows_dropped_earliest": 5, + "rows_per_partition": 37, + "rows_used": 296, + "selected_in_sample_mean": 0.053047748257383835, + "selected_out_of_sample_mean": -0.032240530695887386, + "trials": 4 + } + }, + "with_equalweight_control": { + "mean_S10": { + "combinations": 252, + "degradation_intercept": 0.0008124346550125723, + "degradation_slope": -0.8238587597598237, + "in_sample_selection_counts": [ + 64, + 6, + 71, + 49, + 62 + ], + "lambda_histogram": { + "-0.693147": 73, + "-1.609438": 105, + "0.000000": 32, + "0.693147": 30, + "1.609438": 12 + }, + "lambda_max": 1.6094379124341007, + "lambda_mean": -0.7122343437319405, + "lambda_min": -1.6094379124341005, + "lambda_zero_count": 32, + "metric": "mean", + "partitions": 10, + "pbo": 0.8333333333333334, + "pbo_strict": 0.7063492063492064, + "probability_oos_loss": 0.47619047619047616, + "rows": 301, + "rows_dropped_earliest": 1, + "rows_per_partition": 30, + "rows_used": 300, + "selected_in_sample_mean": 0.0013663870558545004, + "selected_out_of_sample_mean": -0.00031327529017559347, + "trials": 5 + }, + "mean_S12": { + "combinations": 924, + "degradation_intercept": 0.0007711894468638225, + "degradation_slope": -0.7328454819776022, + "in_sample_selection_counts": [ + 156, + 18, + 253, + 172, + 325 + ], + "lambda_histogram": { + "-0.693147": 273, + "-1.609438": 339, + "0.000000": 185, + "0.693147": 85, + "1.609438": 42 + }, + "lambda_max": 1.6094379124341007, + "lambda_mean": -0.6583492748248891, + "lambda_min": -1.6094379124341005, + "lambda_zero_count": 185, + "metric": "mean", + "partitions": 12, + "pbo": 0.8625541125541125, + "pbo_strict": 0.6623376623376623, + "probability_oos_loss": 0.47294372294372294, + "rows": 301, + "rows_dropped_earliest": 1, + "rows_per_partition": 25, + "rows_used": 300, + "selected_in_sample_mean": 0.001423733394545966, + "selected_out_of_sample_mean": -0.0002721871388698237, + "trials": 5 + }, + "mean_S16": { + "combinations": 12870, + "degradation_intercept": 0.0007905055762317812, + "degradation_slope": -0.7793453291223932, + "in_sample_selection_counts": [ + 2543, + 120, + 3643, + 2446, + 4118 + ], + "lambda_histogram": { + "-0.693147": 3097, + "-1.609438": 4861, + "0.000000": 2751, + "0.693147": 1591, + "1.609438": 570 + }, + "lambda_max": 1.6094379124341007, + "lambda_mean": -0.6177138878149186, + "lambda_min": -1.6094379124341005, + "lambda_zero_count": 2751, + "metric": "mean", + "partitions": 16, + "pbo": 0.832090132090132, + "pbo_strict": 0.6183372183372183, + "probability_oos_loss": 0.4575757575757576, + "rows": 301, + "rows_dropped_earliest": 13, + "rows_per_partition": 18, + "rows_used": 288, + "selected_in_sample_mean": 0.0013298401978357331, + "selected_out_of_sample_mean": -0.0002458991704306967, + "trials": 5 + }, + "mean_S8": { + "combinations": 70, + "degradation_intercept": 0.00042937317639973383, + "degradation_slope": -0.6247513283072168, + "in_sample_selection_counts": [ + 8, + 0, + 12, + 25, + 25 + ], + "lambda_histogram": { + "-0.693147": 17, + "-1.609438": 36, + "0.000000": 11, + "0.693147": 6 + }, + "lambda_max": 0.6931471805599452, + "lambda_mean": -0.936634054768386, + "lambda_min": -1.6094379124341005, + "lambda_zero_count": 11, + "metric": "mean", + "partitions": 8, + "pbo": 0.9142857142857143, + "pbo_strict": 0.7571428571428571, + "probability_oos_loss": 0.4857142857142857, + "rows": 301, + "rows_dropped_earliest": 5, + "rows_per_partition": 37, + "rows_used": 296, + "selected_in_sample_mean": 0.0015279743255972446, + "selected_out_of_sample_mean": -0.0005252308131364684, + "trials": 5 + }, + "sharpe_S10": { + "combinations": 252, + "degradation_intercept": 0.028639078022273817, + "degradation_slope": -0.858701567497155, + "in_sample_selection_counts": [ + 68, + 8, + 65, + 49, + 62 + ], + "lambda_histogram": { + "-0.693147": 73, + "-1.609438": 103, + "0.000000": 39, + "0.693147": 29, + "1.609438": 8 + }, + "lambda_max": 1.6094379124341007, + "lambda_mean": -0.7277582445471316, + "lambda_min": -1.6094379124341005, + "lambda_zero_count": 39, + "metric": "sharpe", + "partitions": 10, + "pbo": 0.8531746031746031, + "pbo_strict": 0.6984126984126984, + "probability_oos_loss": 0.4722222222222222, + "rows": 301, + "rows_dropped_earliest": 1, + "rows_per_partition": 30, + "rows_used": 300, + "selected_in_sample_mean": 0.05483720380767608, + "selected_out_of_sample_mean": -0.018449714844538587, + "trials": 5 + }, + "sharpe_S12": { + "combinations": 924, + "degradation_intercept": 0.022452792840967795, + "degradation_slope": -0.6241540598620049, + "in_sample_selection_counts": [ + 180, + 26, + 227, + 172, + 319 + ], + "lambda_histogram": { + "-0.693147": 250, + "-1.609438": 338, + "0.000000": 214, + "0.693147": 78, + "1.609438": 44 + }, + "lambda_max": 1.6094379124341007, + "lambda_mean": -0.6411212784761213, + "lambda_min": -1.6094379124341005, + "lambda_zero_count": 214, + "metric": "sharpe", + "partitions": 12, + "pbo": 0.8679653679653679, + "pbo_strict": 0.6363636363636364, + "probability_oos_loss": 0.4772727272727273, + "rows": 301, + "rows_dropped_earliest": 1, + "rows_per_partition": 25, + "rows_used": 300, + "selected_in_sample_mean": 0.0588850571579611, + "selected_out_of_sample_mean": -0.014300554649379833, + "trials": 5 + }, + "sharpe_S16": { + "combinations": 12870, + "degradation_intercept": 0.02700644710335813, + "degradation_slope": -0.754633031920549, + "in_sample_selection_counts": [ + 2896, + 167, + 3542, + 2446, + 3819 + ], + "lambda_histogram": { + "-0.693147": 3167, + "-1.609438": 4757, + "0.000000": 3198, + "0.693147": 1504, + "1.609438": 244 + }, + "lambda_max": 1.6094379124341007, + "lambda_mean": -0.6539313955000999, + "lambda_min": -1.6094379124341005, + "lambda_zero_count": 3198, + "metric": "sharpe", + "partitions": 16, + "pbo": 0.8641802641802642, + "pbo_strict": 0.6156954156954157, + "probability_oos_loss": 0.46907536907536906, + "rows": 301, + "rows_dropped_earliest": 13, + "rows_per_partition": 18, + "rows_used": 288, + "selected_in_sample_mean": 0.05285136074821765, + "selected_out_of_sample_mean": -0.012876935499196043, + "trials": 5 + }, + "sharpe_S8": { + "combinations": 70, + "degradation_intercept": 0.011697305728640854, + "degradation_slope": -0.6344411567427698, + "in_sample_selection_counts": [ + 8, + 0, + 17, + 25, + 20 + ], + "lambda_histogram": { + "-0.693147": 16, + "-1.609438": 32, + "0.000000": 19, + "0.693147": 3 + }, + "lambda_max": 0.6931471805599452, + "lambda_mean": -0.8644703792167215, + "lambda_min": -1.6094379124341005, + "lambda_zero_count": 19, + "metric": "sharpe", + "partitions": 8, + "pbo": 0.9571428571428572, + "pbo_strict": 0.6857142857142857, + "probability_oos_loss": 0.4857142857142857, + "rows": 301, + "rows_dropped_earliest": 5, + "rows_per_partition": 37, + "rows_used": 296, + "selected_in_sample_mean": 0.05786714389700346, + "selected_out_of_sample_mean": -0.025015991982774335, + "trials": 5 + } + } + }, + "primary": { + "dsr": 0.5520685371554236, + "dsr_selected_trial": "momentum120", + "pbo": 0.7756021756021756, + "pbo_strict": 0.7756021756021756, + "probability_oos_loss": 0.48003108003108 + }, + "walk_forward": { + "chosen_counts": { + "cash": 7, + "momentum120": 4, + "momentum20": 4, + "momentum60": 5 + }, + "first_decision": "2016-01-14", + "folds": 20, + "last_decision": "2020-12-16", + "moments": { + "kurtosis": 17.496936247274043, + "mean": -0.0003171285205987647, + "n": 249, + "skew": -1.812947934946181, + "std": 0.02099941875059171 + }, + "observations": 249, + "psr_vs_zero": 0.40476571908946396, + "sharpe": -0.015101776118914188 + } + }, + "conclusion": [ + "DSR of the best retained selection candidate is 0.552 (momentum120), far below a 0.95 confidence threshold, even though only four trials are deflated; its undeflated PSR(0) is 0.667. Adding the equal-weight control (best: equalweight, DSR 0.575) or restricting to momentum rules (DSR 0.561) does not change this.", + "PBO is 0.776 for the primary set (S=16, mean label): the in-sample best candidate ranks below the out-of-sample median in about three of four CSCV splits (N=4, so no split lands exactly on the median). Every sensitivity cell (S 8 to 16, mean or Sharpe metric, three trial sets) has PBO (lambda <= 0) between 0.76 and 0.96; the strict lambda < 0 share is lower where an odd trial count allows exact median ties (momentum-only N=3: 0.34 to 0.48; with the control N=5: 0.62 to 0.76). The retained degradation_slope values (primary -0.727) are not used as evidence: with the mean metric and equal complementary halves, a trial's in-sample plus out-of-sample mean equals twice its mean over the used rows, so splits selecting the same trial lie on a slope of exactly -1 (momentum120 is selected in 5490 of 12870 splits).", + "The frozen walk-forward selector's development out-of-sample series (249 episodes over 20 folds) has negative mean and Sharpe -0.0151, PSR(0) 0.405. This is consistent with the research-evaluation README, which makes no persistent-advantage claim for the training-selected rule.", + "Conclusion: over these retained candidates the evidence is consistent with selection noise, not skill. The no-skill PBO for four exchangeable trials is about 0.5, so a PBO of 0.776 is also consistent with mild anti-persistence of in-sample ranking; the CSCV splits overlap heavily and PBO carries no p-value, so this reading is descriptive. This is a negative diagnostic for this three-ETF control lane; it neither tunes, promotes nor rejects any future strategy." + ], + "limitations": [ + "Inputs are raw-price five-session labels less a fixed 20bp proxy, not portfolio returns; the Sharpe ratios are per-episode label ratios, not investable or annualized Sharpe ratios.", + "The trial count is only this lane's frozen candidate set (4, sensitivity 3 and 5). Trials run elsewhere in the research program are not counted, so the DSR here is an optimistic upper bound relative to program-wide multiple testing; no effective-N clustering was applied although momentum60 and momentum120 share 60% of development labels.", + "CSCV with three to five highly correlated trials gives only a few distinct relative ranks (four logit values in the primary S=16 run) and assumes the contiguous 18-episode blocks are exchangeable; PBO is descriptive and carries no p-value.", + "The CSCV degradation slope is dominated by a construction identity: with the mean metric and equal complementary halves, in-sample + out-of-sample = 2 x the trial's mean over the used rows, so within one selected trial the slope is exactly -1 and the pooled slope mixes that with the spread of the selected trials' full-sample means. It is retained for completeness and is not evidence about ranking persistence; no between-trial decomposition was computed. With the Sharpe metric the identity is not exact, but the same complementary-half dependence applies.", + "Only development episodes (2015-01-02 to 2020-12-16) are used; the inspected 2021Q1 reserved segment (11 episodes) is excluded and was not re-scored.", + "The earliest T mod S rows (13 at S=16) are dropped; results for other truncation choices were not computed.", + "The DSR worked-example constants are transcribed from the paper rather than re-fetched; the PBO tests are locally constructed synthetic fixtures, not upstream tests.", + "Evidence class is a local integration check with an independent numpy recomputation; no upstream implementation was executed or compared." + ] +} diff --git a/blueprints/us-equities/research-evaluation/README.md b/blueprints/us-equities/research-evaluation/README.md index 95dfdddb8..6b2d6a767 100644 --- a/blueprints/us-equities/research-evaluation/README.md +++ b/blueprints/us-equities/research-evaluation/README.md @@ -76,6 +76,15 @@ Independent review verified all 42 retained artifact hashes, exact public/native result equality, all 21 selection hashes, the pre-scoring plan/source/lock freeze, the complete ledger counts and 12 focused tests without rerunning scoring. +## Overfitting controls — September 24, 2026 + +The [overfitting-controls lane](../overfitting-controls/README.md) computed two +diagnostics offline from this run's retained 6,605-record ledger over 301 +development episodes. For the four selection candidates, the deflated Sharpe +ratio is **0.552** and the CSCV probability of backtest overfitting is +**0.776**. The diagnostics add no skill evidence and change no selection. The +reserved segment was not re-scored. + ## Native adoption and execution Keep the working SDK unchanged. Choose a fresh private `WAVE_DIR`, an installed diff --git a/manifests/evidence.json b/manifests/evidence.json index dbd10bbae..9cd20938c 100644 --- a/manifests/evidence.json +++ b/manifests/evidence.json @@ -8870,8 +8870,8 @@ }, { "path": "blueprints/us-equities/research-evaluation/README.md", - "sha256": "5f6e4b5cc7338e9f5fc3edeb0d2d5092f1c2ba779d0243ed3ba29da76a80edde", - "bytes": 8589 + "sha256": "89b428b4ea8c289472805bb2eb5c45d7dcb0333011ad61097fb0d6bf5aee9d5c", + "bytes": 9054 }, { "path": "blueprints/us-equities/research-evaluation/evaluate.py", diff --git a/tests/test_overfitting_controls.py b/tests/test_overfitting_controls.py new file mode 100644 index 000000000..d78d7d4c3 --- /dev/null +++ b/tests/test_overfitting_controls.py @@ -0,0 +1,204 @@ +"""Deflated Sharpe ratio and CSCV PBO: published example, synthetic fixtures and receipt replay.""" +import importlib.util +import itertools +import json +import math +from pathlib import Path +import random +import statistics +import unittest + +ROOT = Path(__file__).resolve().parents[1] +HERE = ROOT / "blueprints/us-equities/overfitting-controls" + + +def load(name, path): + spec = importlib.util.spec_from_file_location(name, path) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +of = load("overfitting_controls_under_test", HERE / "overfitting.py") + + +def brute_force_pbo(matrix, partitions, metric): + """Independent direct recomputation: concatenate rows, statistics module, explicit ranks.""" + t = len(matrix) + size = t // partitions + used = matrix[t - size * partitions:] + blocks = [used[b * size:(b + 1) * size] for b in range(partitions)] + n = len(matrix[0]) + + def score(rows, c): + col = [r[c] for r in rows] + mean = statistics.fmean(col) + if metric == "mean": + return mean + std = statistics.pstdev(col) + return 0.0 if std == 0 else mean / std + + logits = [] + for chosen in itertools.combinations(range(partitions), partitions // 2): + train = [r for b in chosen for r in blocks[b]] + test = [r for b in range(partitions) if b not in chosen for r in blocks[b]] + is_scores = [score(train, c) for c in range(n)] + oos = [score(test, c) for c in range(n)] + best = max(range(n), key=lambda c: (is_scores[c], -c)) + below = sum(v < oos[best] for v in oos) + equal = sum(v == oos[best] for v in oos) + rank = below + (equal + 1) / 2.0 + omega = rank / (n + 1) + logits.append(math.log(omega / (1 - omega))) + return logits + + +class DeflatedSharpeTests(unittest.TestCase): + def test_published_numerical_example(self): + # Bailey and Lopez de Prado (2014), numerical example: N = 100 independent + # trials, annualized V[SR_n] = 1/2, selected annualized SR = 2.5, T = 1250 + # daily observations (250 per year), skewness -3, kurtosis 10. The paper + # reports SR0 ~= 0.1132 (non-annualized) and DSR ~= 0.9004. + sr0_annual = of.expected_max_sharpe(100, 0.5) + sr0 = sr0_annual / math.sqrt(250) + self.assertAlmostEqual(sr0, 0.1132, places=4) + dsr, benchmark = of.deflated_sharpe(2.5 / math.sqrt(250), 1250, -3.0, 10.0, 100, 0.5 / 250) + self.assertAlmostEqual(benchmark, sr0, places=12) + self.assertAlmostEqual(dsr, 0.9004, places=4) + + def test_psr_reduces_to_normal_case(self): + sr, t = 0.1, 500 + self.assertAlmostEqual(of.probabilistic_sharpe(sr, t, 0.0, 3.0), + statistics.NormalDist().cdf(sr * math.sqrt(t - 1) / math.sqrt(1 + sr * sr / 2)), places=14) + self.assertEqual(of.probabilistic_sharpe(0.2, 100, -1.0, 6.0, 0.2), 0.5) + + def test_negative_skew_and_fat_tails_lower_psr(self): + normal = of.probabilistic_sharpe(0.1, 250, 0.0, 3.0) + self.assertLess(of.probabilistic_sharpe(0.1, 250, -2.0, 3.0), normal) + self.assertLess(of.probabilistic_sharpe(0.1, 250, 0.0, 12.0), normal) + + def test_expected_max_matches_order_statistic_tables(self): + # Expected maximum of N iid standard normals (Harter 1961): N=100 -> 2.50759. + # The paper's approximation is asymptotic; allow 1.5% at N=100. + self.assertAlmostEqual(of.expected_max_sharpe(100, 1.0) / 2.50759, 1.0, delta=0.015) + self.assertEqual(of.expected_max_sharpe(1, 4.0), 0.0) + self.assertGreater(of.expected_max_sharpe(1000, 1.0), of.expected_max_sharpe(100, 1.0)) + with self.assertRaises(ValueError): + of.expected_max_sharpe(0, 1.0) + + def test_more_trials_deflate_more(self): + first, _ = of.deflated_sharpe(0.1, 500, 0.0, 3.0, 2, 0.01) + second, _ = of.deflated_sharpe(0.1, 500, 0.0, 3.0, 50, 0.01) + self.assertLess(second, first) + + def test_moments_and_degenerate_series(self): + m = of.moments([1.0, 2.0, 3.0, 4.0]) + self.assertAlmostEqual(m["std"], statistics.pstdev([1, 2, 3, 4]), places=14) + self.assertAlmostEqual(m["skew"], 0.0, places=14) + self.assertAlmostEqual(m["kurtosis"], 1.64, places=12) + self.assertEqual(of.sharpe([0.0, 0.0, 0.0]), 0.0) + with self.assertRaises(ValueError): + of.sharpe([1.0, 1.0]) + with self.assertRaises(ValueError): + of.moments([1.0]) + + +class PboTests(unittest.TestCase): + def test_reversal_is_fully_overfit(self): + # Two blocks: trial 0 wins block 0, trial 1 wins block 1. Whichever block is + # in-sample, the in-sample winner is last out of sample. + matrix = [[1.0, 0.0], [1.0, 0.0], [0.0, 1.0], [0.0, 1.0]] + result = of.cscv_pbo(matrix, partitions=2) + self.assertEqual(result["combinations"], 2) + self.assertEqual(result["pbo"], 1.0) + self.assertAlmostEqual(result["lambda_max"], math.log(0.5), places=14) + + def test_dominant_trial_is_not_overfit(self): + rng = random.Random(7) + matrix = [[rng.gauss(0, 1) + 10.0] + [rng.gauss(0, 1) for _ in range(5)] for _ in range(160)] + result = of.cscv_pbo(matrix, partitions=8) + self.assertEqual(result["combinations"], math.comb(8, 4)) + self.assertEqual(result["pbo"], 0.0) + self.assertEqual(result["in_sample_selection_counts"][0], result["combinations"]) + self.assertEqual(result["probability_oos_loss"], 0.0) + + def test_pure_noise_is_near_one_half(self): + pbos = [] + for seed in range(20): + rng = random.Random(seed) + matrix = [[rng.gauss(0, 1) for _ in range(10)] for _ in range(200)] + pbos.append(of.cscv_pbo(matrix, partitions=10)["pbo"]) + self.assertAlmostEqual(statistics.fmean(pbos), 0.5, delta=0.1) + + def test_block_merge_equals_direct_recomputation(self): + rng = random.Random(11) + matrix = [[rng.gauss(0.001 * c, 0.02) for c in range(4)] + [0.0] for _ in range(103)] + for metric in ("mean", "sharpe"): + fast = of.cscv_pbo(matrix, partitions=8, metric=metric) + slow = brute_force_pbo(matrix, 8, metric) + self.assertEqual(fast["rows_dropped_earliest"], 103 - 8 * 12) + self.assertEqual(len(slow), fast["combinations"]) + for a, b in zip(fast["logits"], slow): + self.assertAlmostEqual(a, b, places=12) + self.assertEqual(fast["pbo"], sum(x <= 0 for x in slow) / len(slow)) + + def test_ties_use_average_rank_and_first_in_sample(self): + matrix = [[0.0, 0.0, 0.0] for _ in range(8)] + result = of.cscv_pbo(matrix, partitions=4) + self.assertEqual(result["in_sample_selection_counts"], [6, 0, 0]) + self.assertEqual(result["lambda_zero_count"], 6) + self.assertEqual(result["pbo"], 1.0) + self.assertEqual(result["pbo_strict"], 0.0) + + def test_mean_metric_degradation_slope_is_an_identity(self): + # With the mean metric and equal complementary halves, in-sample + out-of-sample + # equals twice the trial's mean over the used rows, so a trial that is always + # selected gives a slope of exactly -1 whatever its skill. + rng = random.Random(7) + matrix = [[rng.gauss(0, 1) + 10.0] + [rng.gauss(0, 1) for _ in range(5)] for _ in range(160)] + result = of.cscv_pbo(matrix, partitions=8) + self.assertEqual(result["pbo"], 0.0) + self.assertAlmostEqual(result["degradation_slope"], -1.0, places=12) + # Mixed selection: the selected in- and out-of-sample means still sum to twice + # the selection-weighted full means. + rng = random.Random(3) + matrix = [[rng.gauss(0.001 * c, 0.02) for c in range(4)] for _ in range(103)] + result = of.cscv_pbo(matrix, partitions=8) + used = matrix[result["rows_dropped_earliest"]:] + full = [statistics.fmean(r[c] for r in used) for c in range(4)] + weighted = math.fsum(k * f for k, f in zip(result["in_sample_selection_counts"], full)) / result["combinations"] + self.assertGreater(sum(k > 0 for k in result["in_sample_selection_counts"]), 1) + self.assertAlmostEqual(result["selected_in_sample_mean"] + result["selected_out_of_sample_mean"], + 2.0 * weighted, places=14) + + def test_rejects_invalid_configuration(self): + for args in ([[1.0, 2.0]] * 8, 3), ([[1.0]] * 8, 2), ([[1.0, 2.0]] * 3, 2), ([[1.0, float("nan")]] * 8, 2): + with self.assertRaises(ValueError): + of.cscv_pbo(*args) + + +class RetainedLedgerReplayTests(unittest.TestCase): + def test_receipt_results_replay_from_retained_ledger(self): + receipt = json.loads((HERE / "receipt.json").read_text()) + runner = load("overfitting_controls_runner", HERE / "evaluate.py") + replay = runner.compute() + + def close(a, b, where): + if isinstance(a, dict): + self.assertEqual(sorted(a), sorted(b), where) + for k in a: + close(a[k], b[k], where + "/" + k) + elif isinstance(a, list): + self.assertEqual(len(a), len(b), where) + for i, (x, y) in enumerate(zip(a, b)): + close(x, y, "%s/%d" % (where, i)) + elif isinstance(a, float) and isinstance(b, float): + self.assertTrue(math.isclose(a, b, rel_tol=1e-9, abs_tol=1e-12), where) + else: + self.assertEqual(a, b, where) + + close(replay, receipt["results"], "results") + + +if __name__ == "__main__": + unittest.main()