Skip to content

Recovery exits each held symbol on its own fresh quote (#215 part 1) - #486

Merged
seathatflowsinourveins merged 2 commits into
mainfrom
claude/recovery-per-symbol-quote-215-20260929
Sep 29, 2026
Merged

seathatflowsinourveins merged 2 commits into
mainfrom
claude/recovery-per-symbol-quote-215-20260929

Conversation

@seathatflowsinourveins

Copy link
Copy Markdown
Owner

Summary

This fixes issue #215, part 1. Before this change, one stale or sparse held symbol could stop recovery from exiting any other symbol. It does not close #215: part 2 (the ambiguous GRML intent with no broker record) and the trial C run-loop stall stay open.

  • recovery.py: each exit gates on its own symbol's quote.
    • The exit takes the alphabetically first held symbol whose own quote is fresh at that turn.
    • A symbol without a fresh quote is deferred, never priced from a stale mark, and never holds back another symbol's exit.
    • Before every exit, recovery checks that a whole exit (2 × order_timeout) still fits before the deadline. Otherwise it ends with recovery_quote_not_fresh if a held symbol is still unquoted, else recovery_deadline_reached.
    • The result reports stale_symbols, which is meaningful only together with recovery_quote_not_fresh.
  • mover_runner.py: the recovery ports' readiness waits on the benchmark quotes only, as the trial port's already does.
    • This covers the forced-recovery port in command_paper and the port in command_recover.
    • recover_mover fails closed with held_symbol_not_subscribed unless every held and unresolved symbol is subscribed.
  • What stays binding: the 3 s quote-age rule for every order, exits included; the kill switch; the loss and drawdown caps; and stale held marks still block entries.
  • Not touched: transport.py, runner.py, safety.py, native_adapter.py and order_contract.py, so the native-fault gate's bound sources do not move.
  • Adaptive lane: runner.py's recovery port still gates start() on every held symbol, until the F1 follow-up. README-recovery.md states this.

Tests (failing-first)

Test On the pre-fix code
PerSymbolRecoveryExits (5 cases): a stale symbol doesn't hold back a fresh one; a stale symbol is exited once it quotes; all-stale waits to the deadline and sends nothing; a late quote starts no exit; no exit starts once a whole exit no longer fits 4 of 5 fail (the late-quote guard passes by design)
MoverPaperCommandWiring.test_recovery_ports_require_only_the_benchmark_quotes [['SPY'],['SPY'],['BBB'],['BBB']] != [['SPY']]*4
RecoveryPortSubscription.test_a_held_or_unresolved_symbol_the_port_does_not_subscribe_fails_closed SafetyError not raised
3 transport guards: an owned exit with a fresh quote reaches the wire under an admission freeze; an exit without one is refused before the wire; start waits on the required basket only pass before and after

Checks at 9d6b873b (rebased on main b0fb65b4)

  • All 20 adaptive-paper modules: 1,203 tests OK, with the pinned runtime (Python 3.12, nautilus-trader 2.0.0rc5, alpaca-py 0.44.0).
  • Source-hash test OK; evidence_manifest --check passed; validate.py passed.
  • trading_gates.py --check passed: native-fault-behaviour stays at bound 6, stale 3 (runner, safety, transport), unchanged from base.
  • git diff --check clean; the three registry tests OK.
  • Review history:
    • an Opus 5.5 evidence-reviewer returned changes-needed on the first revision (1 must-fix, 1 should-fix, 4 nits), and all were repaired;
    • an independent stack-verifier returned pass twice.
  • GPT-6 cross-family review follows before merge.

Evidence class

Local synthetic tests with fake ports and streams only. No broker run, and no paper acceptance for this path. The mover recovery start now depends on a SPY quote within start_timeout (15 s), which is unmeasured on the broker after hours.

SOTA sources

  • alpaca-py v0.44.0 (tag cc4cb3b7, transport.py SDK_COMMIT), alpaca/data/live/websocket.py:
    • :93-99: data_timeout is off by default for legitimately quiet subscriptions;
    • :239-276: staleness is judged per connection, not per symbol.
  • nautilus_trader v2.0.0rc5, crates/risk/src/engine/mod.rs:
    • :434-437 and :2281-2345: a global Halted or Reducing state still admits reduce-only exits, and no quote-age kill condition exists.
    • No upstream Alpaca adapter exists; the engine's native_adapter.py is the adapter.
  • The engine's own rules: README-safety.md (every order needs a quote at most 3 s old); README-transport.md:123-128 (the readiness contract).

🤖 Generated with Claude Code

@seathatflowsinourveins seathatflowsinourveins added the lane:trading Trading lane: sim, paper and live north star label Sep 29, 2026
Scout and others added 2 commits September 29, 2026 00:09
recovery.recover took the alphabetically first held symbol and ended with
recovery_quote_not_fresh after one quote_max_age wait, so one held symbol
without a fresh quote ended recovery before any symbol after it was sold.
Each exit now takes the alphabetically first held symbol whose own quote is
fresh at that turn (not the freshest). A symbol without one is skipped and
exited once it quotes. No exit starts, for a symbol quoted all along as for
one that has just quoted, unless a whole exit (two order timeouts: the fill
wait and a cancel's wait) still fits before the order deadline. The check
runs after every wait, so bounded() cannot cut a late exit after its POST.
When no whole exit fits, recovery ends with recovery_quote_not_fresh if a
held symbol is still unquoted (named in the result's new stale_symbols
field) and otherwise with recovery_deadline_reached. No exit is priced
without its own fresh quote, and a failed exit, including the transport's
pre-wire not_sent refusal, still ends recovery.

mover_runner.py: the forced recovery's port and the recover command's port
require only the benchmark quotes, as a trial's port does, so a sparse held
mover cannot keep the recovery transport from starting. Readiness then no
longer shows that the port subscribes every held symbol, so recover_mover
raises held_symbol_not_subscribed before recovery starts the port when a
held or unresolved symbol is not subscribed.

Tests, written first:
- PerSymbolRecoveryExits (5 tests): 4 fail on d063623 and pass now.
  test_no_exit_starts_once_a_whole_exit_no_longer_fits (the first exit
  fills after the last point a whole exit fits while the other symbol
  freshens during that fill wait) also failed on this change's first
  revision, which started the second exit (['SRZN', 'PFSA'] != ['SRZN']).
- RecoveryPortSubscription (test_adaptive_paper_mover.py): fails on
  d063623 and on the first revision (SafetyError not raised), passes now.
- MoverPaperCommandWiring.test_recovery_ports_require_only_the_benchmark_quotes:
  failed on d063623 (required baskets SPY, SPY, BBB, BBB), passes now.
- Three AsyncTransport guards (an admission freeze does not block an owned
  exit with a fresh quote; an exit without its own fresh quote is refused
  before the wire as not_sent; start() waits on the required basket only)
  pass before and after; transport.py is unchanged.

README-recovery.md scopes the per-symbol handling to the mover's recovery
ports: runner.py's recovery port (the adaptive lane) still gates start() on
every held symbol's quote until the F1 follow-up. It qualifies the exit
order (alphabetical, not the freshest; a pre-wire refusal still ends
recovery), states that stale_symbols is meaningful only together with
recovery_quote_not_fresh, and its Verification section covers the
PerSymbolRecoveryExits cases without stale test counts. README-mover.md
describes the new exit order, the whole-exit stop and the subscription
guard.

source-hashes.json and manifests/evidence.json are re-registered per the
docs/lanes.md hot-file protocol on origin/main; the component matrix and the
new-host grand list regenerate unchanged.

runner.py, transport.py, safety.py, native_adapter.py and order_contract.py
are unchanged. #215 stays open for part 2, the ambiguous GRML intent and
the trial C loop stall.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ce and timing tests (#486 GPT-6 review)

Implemented by GPT-6 (gpt-6-astra, effort max) through codex exec in the owned worktree, test-first;
acceptance re-run by the coordinator before commit.

Co-Authored-By: GPT-6 (Codex) <noreply@openai.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@seathatflowsinourveins
seathatflowsinourveins force-pushed the claude/recovery-per-symbol-quote-215-20260929 branch from 9d6b873 to 88536ec Compare September 29, 2026 04:10
@seathatflowsinourveins
seathatflowsinourveins enabled auto-merge (squash) September 29, 2026 04:10
@seathatflowsinourveins

Copy link
Copy Markdown
Owner Author

Cross-family review round. GPT-6 (gpt-6-astra, effort max, read-only) reviewed head 9d6b873b and returned changes-needed: 1 must-fix and 3 should-fix. All four are repaired in 88536ec1 (rebased on main 1b8d6334). GPT-6 implemented the repair test-first through codex exec in the owned worktree, and the coordinator re-ran the acceptance before committing.

  1. Must-fix: the whole-exit budget is rechecked after position preparation and pre-wire authorization, immediately before submit(). New tests: test_position_preparation_cannot_spend_the_whole_exit_budget, test_pre_wire_authorization_cannot_spend_the_whole_exit_budget and test_pre_wire_authorization_preserves_the_unquoted_error. All three fail on 9d6b873b: the base sends an exit with 0.75 s left against 0.8 s required.
  2. Distinct per-symbol prices: PFSA 10.00/10.01 and SRZN 20.00/20.01. The review's quotes[held[0]] mutant now fails (['9.98'] != ['19.98']).
  3. All-stale then timely arrival: an independent task delivers the quotes before the cutoff. The removed-quote-wait mutant now fails.
  4. Deadline decisions run on a controlled monotonic clock, injected into the recovery module only. The hang watchdog stays on real time. It passes under the reviewer's 550 ms scheduler delay, which failed the old test.

Coordinator acceptance on the pre-rebase tree:

  • all 20 adaptive-paper modules OK, and the recovery module OK 3 times in a row;
  • the source-hash test OK; evidence_manifest --check and validate.py passed;
  • trading_gates.py --check passed, with native-fault bound 6 and stale 3, unchanged;
  • git diff --check clean; the registry tests OK.
    The hot files were re-checked after the rebase. transport.py, runner.py, safety.py, native_adapter.py and order_contract.py are untouched.

That completes one review and one repair per reviewer family. Guarded auto-merge is set at 88536ec1.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lane:trading Trading lane: sim, paper and live north star

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Sparse mover quotes freeze the transport (exits and recovery stop); attempted submit under a frozen transport has no operator resolution

1 participant