Deflated Sharpe and PBO (CSCV) over the 6,605 retained research candidates

blueprints/us-equities/overfitting-controls/README.md

@@ -0,0 +1,96 @@
+# Backtest-overfitting controls
+
+This lane adds two selection-bias diagnostics to the multi-candidate research
+lane and applies them to the retained
+[research-evaluation](../research-evaluation/README.md) candidate ledger:
+
+- **Deflated Sharpe ratio (DSR)**, Bailey and Lopez de Prado (2014): the
+  probabilistic Sharpe ratio of the best trial, measured against the expected
+  maximum Sharpe of N null trials rather than zero.
+- **Probability of backtest overfitting (PBO)** by combinatorially symmetric
+  cross-validation (CSCV), Bailey, Borwein, Lopez de Prado and Zhu (2017): the
+  share of in-sample/out-of-sample block splits in which the in-sample best trial
+  ranks at or below the out-of-sample median.
+
+[`overfitting.py`](overfitting.py) implements both from the papers using only the
+Python standard library. Nothing was copied or vendored. The citations and the
+formulas are in its docstring. [`evaluate.py`](evaluate.py) applies the module
+offline to the hash-pinned ledger using the parameters in
+[`params.json`](params.json). Those parameters were frozen before the first
+computation. [`crosscheck.py`](crosscheck.py) recomputes the primary DSR and PBO
+independently with numpy.
+
+## Result — September 24, 2026
+
+Input: the 6,605-record candidate ledger (SHA-256 `08ea5309…2f42672`, equal to
+the research-evaluation receipt's `ledger_sha256`). The in-tree copy is the
+byte-identical gap-wave2 regeneration. No data was refetched. After
+deduplication, the development segment has 301 aligned five-session episodes per
+candidate, from 2015-01-02 to 2020-12-16. The reserved 2021Q1 segment is
+excluded.
+
+| Diagnostic (frozen four-candidate selection set) | Value |
+| --- | ---: |
+| Best per-episode Sharpe (momentum120), not annualized | 0.0254 |
+| Expected maximum null Sharpe SR0 (N = 4) | 0.0177 |
+| PSR against zero | 0.667 |
+| **Deflated Sharpe ratio** | **0.552** |
+| **PBO** (S = 16, mean-label metric, 12,870 splits) | **0.776** |
+| Probability that the selected trial loses out of sample | 0.480 |
+| Walk-forward selected series: Sharpe / PSR(0), 249 episodes | -0.0151 / 0.405 |
+
+All sensitivity cells give PBO (lambda <= 0) between 0.76 and 0.96. The cells
+cover S = 8, 10, 12 and 16, the mean and Sharpe metrics, and three trial sets:
+the primary four, adding the equal-weight control, and momentum rules only.
+DSR stays between 0.55 and 0.58. The [receipt](receipt.json) retains every cell,
+the strict lambda < 0 shares, input and source hashes, commands and the
+independent recomputation.
+
+The receipt also keeps the paper's degradation slope (the selected trial's
+out-of-sample metric regressed on its in-sample metric; -0.727 in the primary
+cell). It is not used as evidence. With the mean metric and equal complementary
+halves, each trial's in-sample mean plus its out-of-sample mean equals twice its
+mean over all used rows. Every split that selects the same trial therefore lies
+on a line of slope exactly -1, and momentum120 is selected in 5,490 of the
+12,870 splits. The pooled slope is pulled towards -1 by construction, and it
+says almost nothing about whether rankings persist.
+
+**Conclusion.** These retained candidates show no deflated-Sharpe evidence of
+skill. The in-sample best candidate usually ranks below the out-of-sample
+median. With four trials and no skill, PBO would be about 0.5, so 0.776 is
+consistent with selection noise or mild anti-persistence of in-sample ranking,
+not skill. The CSCV splits overlap heavily and PBO has no p-value, so this
+reading is descriptive. This negative diagnostic covers only this three-ETF
+control lane. It does not tune, promote or reject any future strategy.
+
+## Boundaries
+
+- The inputs are raw-price five-session labels less a fixed 20bp proxy, not
+  portfolio returns. The Sharpe values are per-episode label ratios.
+- N counts only this lane's frozen candidates. The research program ran other
+  trials, so relative to all of them this DSR is optimistic. No effective-N
+  clustering was applied. momentum60 and momentum120 share about 60% of their
+  development labels.
+- With three to five correlated trials, CSCV has only a few distinct ranks.
+  Contiguous 18-episode blocks are assumed exchangeable. PBO is descriptive and
+  has no p-value.
+- Evidence class: a local integration check with an independent numpy
+  recomputation. The DSR unit test reproduces the paper's numerical example
+  (SR0 0.1132, DSR 0.9004). Those constants are transcribed from the paper,
+  which was not re-fetched offline. The PBO tests are locally constructed
+  synthetic fixtures. No upstream implementation was executed.
+
+## Reproduce
+
+```sh
+python3 blueprints/us-equities/overfitting-controls/evaluate.py
+python3 -m unittest tests.test_overfitting_controls -v
+# optional independent recomputation (needs numpy)
+python blueprints/us-equities/overfitting-controls/crosscheck.py
+```
+
+Reusing the module for another candidate ledger: pass a T x N matrix of
+per-period trial performance to `cscv_pbo`. Pass the selected trial's Sharpe,
+moments, trial count and cross-trial Sharpe variance to `deflated_sharpe`.
+Freeze the trial set and the parameters before looking at the results. Count
+every trial that was actually tried.

blueprints/us-equities/overfitting-controls/crosscheck.py

@@ -0,0 +1,43 @@
+"""Independent recomputation with numpy (different code path from overfitting.py); normal quantiles from statistics.NormalDist, CDF from math.erf."""
+import json, sys, itertools, math
+from decimal import Decimal
+import numpy as np
+from statistics import NormalDist
+ppf = NormalDist().inv_cdf
+cdf = lambda z: 0.5 * (1 + math.erf(z / math.sqrt(2)))
+
+def rankdata(a):
+    a = np.asarray(a); r = np.empty(len(a))
+    for i, x in enumerate(a):
+        r[i] = (a < x).sum() + ((a == x).sum() + 1) / 2
+    return r
+
+root = sys.argv[1] if len(sys.argv) > 1 else "."
+ledger = json.load(open(root + "/evidence/artifacts/gap-wave2-20260923/us-equities__portfolio-risk/raw/3/regenerated/run-1/candidate-ledger.json"))
+trials = ["momentum20", "momentum60", "momentum120", "cash"]
+v = {}
+for r in ledger:
+    if r["status"] == "observed" and r["decision"] <= "2020-12-31":
+        v[(r["candidate"], r["decision"])] = float(Decimal(r["net_proxy"]))
+dates = sorted({d for c, d in v})
+M = np.array([[v[(c, d)] for c in trials] for d in dates])
+sd = M.std(axis=0)
+sr = np.where(sd > 0, M.mean(axis=0) / np.where(sd > 0, sd, 1), 0.0)
+b = int(np.argmax(sr)); x = M[:, b]; T = len(x)
+d = x - x.mean(); g3 = (d**3).mean() / (d**2).mean()**1.5; g4 = (d**4).mean() / (d**2).mean()**2
+V = np.var(sr, ddof=1); N = len(trials); g = np.euler_gamma
+sr0 = math.sqrt(V) * ((1 - g) * ppf(1 - 1 / N) + g * ppf(1 - 1 / (N * np.e)))
+dsr = cdf((sr[b] - sr0) * math.sqrt(T - 1) / math.sqrt(1 - g3 * sr[b] + (g4 - 1) / 4 * sr[b] ** 2))
+# PBO, S=16, mean metric, direct slicing
+S = 16; size = T // S; U = M[T - size * S:]; blocks = np.split(U, S)
+lam = []
+for J in itertools.combinations(range(S), S // 2):
+    tr = np.vstack([blocks[i] for i in J]); te = np.vstack([blocks[i] for i in range(S) if i not in J])
+    ism, osm = tr.mean(0), te.mean(0)
+    n = int(np.flatnonzero(ism == ism.max())[0])
+    w = rankdata(osm)[n] / (len(trials) + 1)
+    lam.append(math.log(w / (1 - w)))
+lam = np.array(lam)
+print(json.dumps({"T": T, "selected": trials[b], "sr": sr[b], "sr0": sr0, "dsr": dsr,
+                  "pbo": float((lam <= 0).mean()), "combinations": len(lam),
+                  "numpy": np.__version__}, indent=1))

blueprints/us-equities/overfitting-controls/evaluate.py

@@ -0,0 +1,165 @@
+#!/usr/bin/env python3
+"""Deflated Sharpe ratio and CSCV PBO over the retained research-evaluation ledger.
+
+Offline: reads only the hash-pinned ledger, the research-evaluation receipt and
+params.json. Prints deterministic results JSON (no timestamps) to stdout.
+"""
+import argparse
+from decimal import Decimal
+import hashlib
+import importlib.util
+import json
+import math
+from pathlib import Path
+
+HERE = Path(__file__).resolve().parent
+ROOT = HERE.parents[2]
+
+
+def _library():
+    spec = importlib.util.spec_from_file_location("overfitting_controls", HERE / "overfitting.py")
+    module = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(module)
+    return module
+
+
+of = _library()
+
+
+def sha256(path):
+    return hashlib.sha256(Path(path).read_bytes()).hexdigest()
+
+
+def pointer(document, path):
+    node = document
+    for token in path.strip("/").split("/"):
+        node = node[int(token)] if isinstance(node, list) else node[token]
+    return node
+
+
+def load_inputs(root, params):
+    src = params["source"]
+    ledger_path, receipt_path = root / src["ledger_path"], root / src["research_receipt_path"]
+    if sha256(ledger_path) != src["ledger_sha256"]:
+        raise ValueError("ledger hash mismatch")
+    receipt = json.loads(receipt_path.read_text())
+    if pointer(receipt, src["research_receipt_ledger_pointer"]) != src["ledger_sha256"]:
+        raise ValueError("research receipt does not name this ledger")
+    ledger = json.loads(ledger_path.read_text())
+    if len(ledger) != src["ledger_records"]:
+        raise ValueError("ledger record count mismatch")
+    return ledger, receipt
+
+
+def episode_values(ledger, development_end):
+    """One net_proxy per (candidate, decision) in the development segment; conflicts are errors."""
+    values = {}
+    for row in ledger:
+        if row["status"] != "observed" or row["decision"] > development_end:
+            continue
+        key = (row["candidate"], row["decision"])
+        value = Decimal(row["net_proxy"])
+        if key in values and values[key] != value:
+            raise ValueError("conflicting duplicate episode label: %s %s" % key)
+        values[key] = value
+    return values
+
+
+def aligned_matrix(values, trials):
+    dates = sorted({d for (c, d) in values if c in trials})
+    missing = [(c, d) for c in trials for d in dates if (c, d) not in values]
+    if missing or not dates:
+        raise ValueError("trial series are not aligned")
+    return dates, [[float(values[(c, d)]) for c in trials] for d in dates]
+
+
+def dsr_block(trials, matrix):
+    columns = list(zip(*matrix))
+    per_trial = {}
+    for name, column in zip(trials, columns):
+        m = of.moments(column)
+        per_trial[name] = dict(m, sharpe=of.sharpe_from(m["mean"], m["std"]))
+    srs = [per_trial[t]["sharpe"] for t in trials]
+    best = max(range(len(trials)), key=lambda i: (srs[i], -i))
+    selected = per_trial[trials[best]]
+    mean_sr = math.fsum(srs) / len(srs)
+    variance = math.fsum((s - mean_sr) ** 2 for s in srs) / (len(srs) - 1)
+    dsr, sr0 = of.deflated_sharpe(selected["sharpe"], selected["n"], selected["skew"],
+                                  selected["kurtosis"], len(trials), variance)
+    psr0 = of.probabilistic_sharpe(selected["sharpe"], selected["n"], selected["skew"], selected["kurtosis"], 0.0)
+    return {"trials": list(trials), "observations": len(matrix), "per_trial": per_trial,
+            "selected_trial": trials[best], "selected_sharpe": selected["sharpe"],
+            "trial_sharpe_variance_ddof1": variance, "expected_max_sharpe_sr0": sr0,
+            "psr_vs_zero": psr0, "dsr": dsr}
+
+
+def pbo_block(matrix, partitions, metric):
+    result = of.cscv_pbo(matrix, partitions, metric)
+    logits = result.pop("logits")
+    histogram = {}
+    for x in logits:
+        key = "%.6f" % x
+        histogram[key] = histogram.get(key, 0) + 1
+    result["lambda_histogram"] = dict(sorted(histogram.items(), key=lambda kv: float(kv[0])))
+    return result
+
+
+def walk_forward_block(ledger, receipt):
+    folds = receipt["results"]["development_folds"]
+    chosen = {f["id"]: f["chosen"] for f in folds}
+    series = sorted((r["decision"], float(Decimal(r["net_proxy"])), r["fold"])
+                    for r in ledger if r["phase"] == "evaluation" and r["status"] == "observed"
+                    and r["fold"] in chosen and r["candidate"] == chosen[r["fold"]])
+    decisions = [d for d, _, _ in series]
+    if len(set(decisions)) != len(decisions):
+        raise ValueError("overlapping walk-forward evaluation episodes")
+    values = [v for _, v, _ in series]
+    m = of.moments(values)
+    sr = of.sharpe_from(m["mean"], m["std"])
+    return {"folds": len(chosen), "chosen_counts": {c: list(chosen.values()).count(c) for c in sorted(set(chosen.values()))},
+            "observations": len(values), "first_decision": decisions[0], "last_decision": decisions[-1],
+            "moments": m, "sharpe": sr, "psr_vs_zero": of.probabilistic_sharpe(sr, m["n"], m["skew"], m["kurtosis"], 0.0)}
+
+
+def compute(root=ROOT, params_path=HERE / "params.json"):
+    params = json.loads(Path(params_path).read_text())
+    ledger, receipt = load_inputs(root, params)
+    values = episode_values(ledger, params["series"]["development_end"])
+    dsr, pbo = {}, {}
+    grid = [params["pbo"]["partitions_primary"]] + params["pbo"]["partitions_sensitivity"]
+    metrics = [params["pbo"]["metric_primary"]] + params["pbo"]["metric_sensitivity"]
+    dates = None
+    for name, trials in params["trial_sets"].items():
+        set_dates, matrix = aligned_matrix(values, trials)
+        if dates is None:
+            dates = set_dates
+        elif dates != set_dates:
+            raise ValueError("trial sets cover different decision dates")
+        dsr[name] = dsr_block(trials, matrix)
+        pbo[name] = {"%s_S%d" % (metric, s): pbo_block(matrix, s, metric) for metric in metrics for s in grid}
+    primary = pbo["primary"]["%s_S%d" % (params["pbo"]["metric_primary"], params["pbo"]["partitions_primary"])]
+    return {
+        "inputs": {"params_sha256": sha256(params_path),
+                   "ledger_sha256": sha256(root / params["source"]["ledger_path"]),
+                   "research_receipt_sha256": sha256(root / params["source"]["research_receipt_path"]),
+                   "overfitting_py_sha256": sha256(HERE / "overfitting.py"),
+                   "evaluate_py_sha256": sha256(HERE / "evaluate.py")},
+        "ledger_records": len(ledger),
+        "development_episodes": len(dates), "first_decision": dates[0], "last_decision": dates[-1],
+        "dsr": dsr, "pbo": pbo,
+        "primary": {"dsr": dsr["primary"]["dsr"], "dsr_selected_trial": dsr["primary"]["selected_trial"],
+                    "pbo": primary["pbo"], "pbo_strict": primary["pbo_strict"],
+                    "probability_oos_loss": primary["probability_oos_loss"]},
+        "walk_forward": walk_forward_block(ledger, receipt),
+    }
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--root", type=Path, default=ROOT)
+    args = parser.parse_args()
+    print(json.dumps(compute(args.root.resolve()), indent=2, sort_keys=True))
+
+
+if __name__ == "__main__":
+    main()