Skip to content

Live rung: strategy out-of-sample holdout, then separate paper performance, and the standing live-readiness instruction - #183

Closed
seathatflowsinourveins wants to merge 1 commit into
mainfrom
claude/live-performance-gate-20260924
Closed

seathatflowsinourveins wants to merge 1 commit into
mainfrom
claude/live-performance-gate-20260924

Conversation

@seathatflowsinourveins

@seathatflowsinourveins seathatflowsinourveins commented Sep 24, 2026 •

Copy link
Copy Markdown
Collaborator

What

Adds two required live-rung gates to catalogs/us-equities/gates-20260922.json. Their criteria are fixed before any strategy qualifies. They were revised before merge after an independent review (four majors, five minors). strategy-historical-holdout is renamed and redesigned as strategy-out-of-sample-holdout.

strategy-out-of-sample-holdout

Established by one of two routes:

  • (i) Historical holdout. The strategy's frozen protocol passes its own holdout rule, including that protocol's minimum trades and multiple-testing adjustment. Any lower bound is computed at the family-wise adjusted level.
    • For mover-early-entry v1 (protocol.json selection_and_pass_criteria.holdout), that means at most 5 rule-exits, at least 50 trades each, and a Holm-adjusted one-sided p ≤ 0.05.
    • The paper candidate is the one named by paper_e2e.
  • (ii) Preregistered prospective paper holdout. A forward paper study passes its own preregistered test.
    • Its protocol, configs and code are pinned by sha256 before the first order.
    • It is analysed exactly once, at its preregistered analysis point.
    • For the shape, see incentive-monitor/forward-protocol-v1.json on claude/incentive-monitor-20260924.

Both routes need:

  • at least 100 round trips and 20 sessions, or the protocol's larger minimums;
  • returns net of costs from a cost table and a fee schedule, each pinned by sha256. The current instances are cost-table-run-v1.json be50cbdf… and fees.json fa1f468b…. Cost per side is the quoted half-spread plus the protocol's slippage rule;
  • recorded market data or real Alpaca paper fills, pinned by sha256. Synthetic, offline or fixture evidence never counts.

strategy-paper-performance

This gate needs a separate, later paper period:

  • the same strategy and the same frozen configs, pinned by sha256;
  • starting after the out-of-sample gate's analysis point, so no session counts toward both gates.

Pass criteria:

  • Analysis point: one, preregistered. It is the first session end with ≥ 20 sessions and ≥ 100 round trips. It is analysed once, with no repeated testing.
  • Estimator: pooled mean net return per round trip. Fees come from the pinned fee schedule. Alpaca paper charges no commission, and the spread is in the fills.
  • Lower bound: a one-sided 95% circular 5-session block bootstrap lower bound > 0.
  • Drawdown: max drawdown of the strategy's paper equity ≤ 20%.
  • Receipts: zero needs_attention.
  • Slippage: adverse slippage is measured per side, in bps: entry (fill/model − 1) and exit (model/fill − 1). Its median must not exceed the pinned per-side cost assumption. With a bucketed table, the median of each fill's excess over its own bucket's cost must be ≤ 0.
    • The comparator is the strategy's own paper-vs-model script, named in its protocol.
    • For mover, that is paper_compare.py. Its exit_diff_bps uses the opposite sign, so the adverse exit value is recomputed rather than copied.
  • Sessions: a session is a scheduled trading day of the strategy. Days without trades count.
  • Leverage: leverage above 1x counts only at an established ladder rung.

Receipt and checker

Both gates share blueprints/us-equities/strategy-performance/receipt.json, read at /out_of_sample/status and /paper_performance/status.

The notes preregister that both sections carry the same strategy_id, protocol_sha256 and config_sha256. The checker verifies only the status strings, and a reviewer verifies the rest before a dated flip commit.

live-go note

Readiness is now stated as: the live lane is ready for the user's decision when the checker's blocking.live equals ["live-go"]. The user's 2026-09-24 instruction stays recorded:

  • paper and simulation are not subject to human approval steps, and their rung gates still apply;
  • the user stores the live key and gives the explicit go;
  • automation never flips live-go or places live orders.

"Never gated" is removed.

Handbook

Tests

  • tests/test_trading_gates.py adds both gates to the GOOD/BAD receipt fixtures.
  • A new test checks that a receipt holding only the out_of_sample section flips only that gate.

Evidence (rebased on origin/main 5af3581, raw)

  • python3 scripts/trading_gates.py --check: passed. 22 gates; rung_ready is sim true, paper true, live false. blocking.live lists the five open live gates, both strategy gates and live-go. There are no flip candidates.
  • python3 scripts/evidence_manifest.py --check: passed (5,235 files).
  • python3 scripts/validate.py: passed (5,235 hashed files, 143 receipts).
  • python3 -m unittest tests.test_trading_gates: 19 tests OK.
  • Full python3 -m unittest under ecosystem-bounded-run: Ran 3945 tests, OK (skipped=419).
  • Guarded gitleaks git --log-opts=origin/main..HEAD --no-banner --redact: 1 commit scanned, no leaks found.

These are synthetic fixtures and local integration checks. No receipt exists at the shared path.

Status

No strategy qualifies today: mover v1 and v2 had no development pass (#162). The prospective route is not yet on main: the incentive-monitor study proposes it on its own branch, and no gate status follows from it until this PR merges.

🤖 Generated with Claude Code

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

…mance, and the standing live-readiness instruction

Two required live gates, criteria fixed before any strategy qualifies and
revised before merge after an independent review.

strategy-out-of-sample-holdout is established by ONE of two routes: (i) the
strategy's frozen protocol passes its own holdout rule, with its minimum trades
and multiple-testing adjustment and any lower bound at the family-wise adjusted
level (mover-early-entry v1: <= 5 rule-exits, >= 50 trades each, Holm p <= 0.05,
paper candidate per paper_e2e); or (ii) a preregistered prospective paper study,
pinned by sha256 before its first order, passes its own test at its single
analysis point. Both need >= 100 round trips and 20 sessions (or the protocol's
larger minimums), costs from a sha256-pinned cost table (per-side half-spread
plus the protocol's slippage rule) and fee schedule, and recorded market data or
real Alpaca paper fills pinned by sha256; synthetic, offline or fixture evidence
never counts.

strategy-paper-performance needs a separate, later paper period with the same
frozen configs (no session counts toward both gates), one preregistered analysis
point (first session end with >= 20 sessions and >= 100 round trips, analysed
once), a pooled per-round-trip net estimator with a one-sided 95% circular
5-session block bootstrap lower bound > 0, drawdown <= 20%, zero needs_attention,
per-side adverse slippage (entry fill/model - 1, exit model/fill - 1) with a
median within the pinned cost assumption via the strategy's own comparator,
scheduled trading days as sessions, and leverage above 1x only at an established
ladder rung.

Both gates share blueprints/us-equities/strategy-performance/receipt.json; its
sections must carry the same strategy_id, protocol_sha256 and config_sha256, and
the checker verifies only the status strings. The live-go note records the
user's 2026-09-24 instruction: paper and simulation are not subject to human
approval steps; the live lane is ready for the user's decision when the checker's
blocking.live equals ["live-go"]; the user stores the live key and gives the
explicit go; automation never flips live-go or places live orders. The handbook
table is dated September 24 (22 gates, 21 checked) and its paper row matches the
checker. Tests add both gates to the GOOD/BAD receipt fixtures and check that an
out_of_sample-only receipt flips only that gate. No strategy qualifies today
(#162).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@seathatflowsinourveins seathatflowsinourveins changed the title Live rung: strategy performance gates (holdout, then paper) and the standing live-readiness instruction Live rung: strategy out-of-sample holdout, then separate paper performance, and the standing live-readiness instruction Sep 24, 2026
@seathatflowsinourveins
seathatflowsinourveins force-pushed the claude/live-performance-gate-20260924 branch from 320fe65 to 548246f Compare September 24, 2026 14:58
@seathatflowsinourveins seathatflowsinourveins added the lane:trading Trading lane: sim, paper and live north star label Sep 25, 2026
@seathatflowsinourveins

Copy link
Copy Markdown
Collaborator Author

Closing: this PR adds required live-rung gates and a standing live-readiness instruction. Live trading is out of scope for this repository; the owner's rule is paper lane only here, with live handled elsewhere. The branch is kept, so the gate criteria can be reopened or carried to the private trading repository. Part of the 2026-09-25 noise cleanup approved by the repository owner; the trading-lane session acknowledged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lane:trading Trading lane: sim, paper and live north star

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant