Run the mover paper fixtures on a fixed clock, out of the overnight session

[seathatflowsinourveins]

Run the mover paper fixtures on a fixed clock, so the overnight session gate cannot fail them. Test-only: no product source changes.

Cause

From 20:00 ET on Sunday 2026-10-11 (00:00Z), 10 tests failed on main (791eab026) when run locally with the pinned native runtime, in a network-less jail with TZ=UTC:

• tests.test_adaptive_paper_mover_native: 7 of 28
• tests.test_alpaca_admission_regressions: 1 of 18
• tests.test_adaptive_paper_fees: 2 of 61

The failing assertions are 'completed_no_signals' != 'passed' and its tuple forms. On 3a8a8461e, which adds #1015's test_sell_age_reaches_paper_and_forced_recovery_without_widening_entries, the counts at the real clock (OVERNIGHT) are 7 of 28, 2 of 19 and 2 of 61.

runner.Controller._check_overnight_status (runner.py:934-958 at 3a8a8461e) classifies the controller's own clock with sessions.session_at (sessions.py:126-172). That clock defaults to the time.time builtin, bound at import (runner.py:828). In the OVERNIGHT session, 20:00 ET on the prior civil day to 04:00 ET of an XNYS trade date, the gate refuses every order on an extended-hours port that lacks an Alpaca overnight asset status. No fixture supplies one. So every entry was refused, the trial ended completed_no_signals (runner._run_native_status), and no residual was left for the forced recovery that several tests expect.

Boundary, measured by moving the process clock to 16 instants (real clock plus an offset, applied to time.time, time.time_ns and the clock= defaults; main 791eab026; failures per module):

Instant	Session	mover_native /28	admission /18	fees /61
Fri 10-09 15:00 ET	RTH	0	0	0
Sat 10-10 12:00 ET	CLOSED	0	0	0
Sun 10-11 19:55 ET (23:55Z)	CLOSED	0	0	0
Sun 10-11 20:05 ET (00:05Z)	OVERNIGHT	7	1	2
Mon 10-12 00:00:01Z	OVERNIGHT	7	1	2
Mon 10-12 00:30Z	OVERNIGHT	7	1	2
Mon 10-12 03:59 ET	OVERNIGHT	1 (crosses 04:00 during the run)	1	2
Mon 10-12 04:00 ET	PRE	0	0	0
Mon 10-12 09:30 ET	RTH	0	0	0
Mon 10-12 16:00 ET	POST	0	0	0
Tue 10-13 19:59 ET	POST	6 (crosses 20:00 during the run)	0	0
Tue 10-13 20:01 ET	OVERNIGHT	7	1	2
Fri 10-09 20:01 ET (Sat 00:01Z)	CLOSED	0	0	0
Wed 11-04 19:30 EST (Thu 00:30Z)	POST	0	0	0
Wed 11-04 20:30 EST (01:30Z)	OVERNIGHT	7	1	2
Wed 11-25 20:30 EST (Thanksgiving eve)	CLOSED	0	0	0

The cause is the ET session and the trade calendar, not the UTC date. CI never ran these tests: validate.yml installs only .github/requirements-ci.txt, which has no NautilusTrader or alpaca-py, so the native classes are skipped. In #1025's job logs, shard 3 skipped all 28 mover_native tests, shard 6 skipped 15 of 18 admission tests and shard 2 skipped 23 of 61 fees tests.

Fix

• tests/adaptive_paper_hermetic.py adds FixtureClock.
  • It moves the wall clock to Wednesday 2026-10-07 10:00 ET (RTH). That is later than every instant the fixtures embed: the mover scan day 2026-09-24 and the legacy fee window 2026-09-29T20:25:19Z.
  • It still advances in real time, and one offset per test keeps a test's successive commands on one timeline.
  • patched() installs it as time.time and time.time_ns, and as the clock= default of runner.Controller and mover_strategy.MoverStrategy. Those defaults bound the builtin at import, so a plain time.time patch misses them.
• The controller's clock cannot move alone: the ledger compares it with every quote's timestamp (RiskLimits.quote_max_age_seconds).
• MoverNativeEndToEnd.run_trial, MoverPaperCommandWiring.run_paper (also used by the admission regressions) and MoverLaneFees.run_command run on it.
• The reads that fed those commands from outside now use the same clock: the halt seed's halted_at, a resumption status's datetime.now, post_fees, the checkpoint seam and the request-budget probe.
• run_trial falls back to a fresh clock for test_mover_exit_chain, which calls it with its own TestCase.
• The source-hashes.json rows of the three test files, and the manifests/evidence.json entries of those files and of source-hashes.json, were rewritten with host_receipts.register_file.
• This change does not patch NautilusTrader's own clock; the fixtures already pin the native dispatch clock (native_pre_dispatch).

Red/green evidence

• Fail-first: three tests run a trial, a paper command and a fees trial inside a clock pinned at Sunday 2026-10-11 20:05 ET, asserted OVERNIGHT:
  • MoverNativeEndToEnd.test_an_extended_hours_trial_keeps_the_fixture_clock_after_20_et
  • MoverPaperCommandWiring.test_the_paper_command_keeps_the_fixture_clock_after_20_et
  • MoverLaneFees.test_a_mover_trial_keeps_the_fixture_clock_after_20_et
• Red: with main's runners (the fix reverted, the new tests kept), all three fail with completed_no_signals. That held at the real clock (00:48Z and 02:14Z, OVERNIGHT) and at instants in RTH (Fri 15:00 ET, Mon 09:30 ET), CLOSED (Sun 19:55 ET) and OVERNIGHT (Sun 20:05 ET, Mon 00:30Z, Tue 20:01 ET).
• Green:
  • First base (c735e8b2e), real clock 02:14Z (OVERNIGHT): mover_native 31/31, admission 18/18, fees 62/62, exit_chain 14/14. Main at the same moment still failed 7, 1 and 2 of its tests.
  • Rebased commit (base 3a8a8461e), real clock 03:13Z to 03:15Z (OVERNIGHT): admission 19/19, fees 62/62, exit_chain 14/14, the three binding modules 25/25, scripts/validate.py and scripts/final_stack.py --check exit 0; mover_native 31/31 apart from the x2 intermittent below. Main (3a8a8461e) in the same minutes: mover_native 7 of 28, admission 2 of 19 (both in CashContinuity; Bound paper sell quote age independently of entry freshness #1015 added one of them) and fees 2 of 61 failing. At Mon 09:30 ET, Mon 16:00 ET and Tue 20:01 ET: admission 19/19 and fees 62/62 each, mover_native 31/31 at the first and third and 30/31 (x2) at the second.
  • mover_native, admission and exit_chain pass at Fri 15:00 ET, Sun 19:55 ET, Sun 20:05 ET, Mon 00:30Z and Tue 20:01 ET. fees passes at all 16 instants, and all four modules passed at all 16 before the positive test below was added.
  • mover_native, admission and fees are also green with the jail's zone set to America/New_York.
  • With the fixture start moved 368 and 732 days behind the real clock, mover_native, admission and exit_chain are green. The only fees failure there is the embedded-date case explained under Fix; the real start sits after that window.

Positive overnight test

MoverNativeEndToEnd.test_the_mover_controller_admits_an_overnight_entry_only_with_an_eligible_asset_status keeps the gate itself covered on the mover path.

• It uses a real Ledger and MoverController on the existing clock= seam at Sunday 20:05 ET (asserted OVERNIGHT).
• Without a carrier, before_submit refuses the entry (overnight_status_unknown) and nothing is reserved.
• After trading_status(transport.normalize_overnight_status(...)) with overnight_tradable, the buy is reserved.
• It complements tests/test_adaptive_paper_overnight.py:154-276, which covers the base Controller with a mock ledger.
• An end-to-end overnight trial is not possible on this path: native_adapter.py:767-784 refuses OVERNIGHT dispatch by design.

Checks

Check	Result	Classification
tests.test_adaptive_paper_source_hashes_manifest, tests.test_evidence_manifest, tests.test_trading_gates_bindings (rebased commit)	25/25	green
python3 scripts/validate.py (rebased commit)	exit 0, "status": "passed"	green
python3 scripts/final_stack.py --check (rebased commit)	exit 0 (report-only flags)	green; it was red on the old base until #1025
mover_native, admission, fees, exit_chain at 16 instants and the real clock	green	fixed by this PR
Credential-permission and non-Git-tree tests in the jail (market_research, runner, credential_race, credential_source, alpaca_history_fixture, order_throughput, adaptive_paper_live_manifest, adaptive_paper_sessions, adaptive_paper_transport, ibkr_acceptance, native_faults_min)	same failures as main at every instant	inherited (sandbox artifacts, unchanged files)
test_x2_exits_with_a_partial_entry_timeout_and_an_unfilled_entry	intermittent on main and with this PR	inherited (see below)
AdaptiveLaneFees.test_full_next_trial_budget_prevents_the_http_read	load-sensitive on both trees (class untouched)	inherited (see below)
CI (draft PR, head 9514a87fe, read 05:27:55Z)	28 passed, 5 skipped (the SARIF-upload and zizmor jobs do not run on a PR): validate, validate-shard-0 to validate-shard-7, verdict-review-gate, sota-sources, secret-scan, the CodeQL analyses and the rest	green; the native classes this PR fixes are skipped in CI, so a local run with the pinned native runtime is their evidence

Open intermittents (both predate this PR)

test_x2_exits_with_a_partial_entry_timeout_and_an_unfilled_entry

Builder's runs (2026-10-12 02:41-02:50Z, on the first base): 40 per tree for each kind, main and this PR alternating in one job list, 20 jails at a time, each with its own scratch directory; host 1-minute load average 28.6 to 58.5.

Runs	main	this PR
the test alone, 40 per tree	0 of 40 fail	0 of 40 fail
the whole module, 40 per tree	x2 fails in 5 of 40	x2 fails in 2 of 40 (and test_periodic_reconciliation_with_snapshot_latency_while_holding in 1 of 40)

The failure exists on main at least as often, so this PR's clock does not cause it.

Checks on the rebased commit (03:13Z to 03:26Z, same jail, 8 to 12 jails at a time, host 1-minute load average 28 to 51, real clock OVERNIGHT unless stated): four whole-module runs of this PR's commit at four clocks, in which x2 failed twice (the real clock and Mon 16:00 ET) and every other test passed; then an interleaved A/B of 24 whole-module runs per tree at the real clock: this PR's commit 24 of 24 runs fully green (31/31, no x2 failure), main 3a8a8461e 24 of 24 runs with exactly its 7 deterministic overnight failures and no x2 failure. Pooled with the builder's runs, whole-module x2 failures are 4 of 98 runs on this PR's tree and 5 of 95 on main; an independent reader's whole-module runs gave 0 of 30 on each tree. The runs do not separate the trees. R1 below names the experiment that would.

The receipts of failing runs on both trees show the same signature: an exit order is filled in the ledger but carries no native fill event, and the receipt's native_fill_events count is short. That fits run_mover (mover_runner.py:381-382) ending once the ledger is flat, before the last exit's native OrderFilled reaches the strategy. The engine log at INFO does not record order events (log_events=False), so the ordering itself is unconfirmed. It is a separate, pre-existing issue and is left unchanged here.

AdaptiveLaneFees.test_full_next_trial_budget_prevents_the_http_read

An independent reader saw tests/test_adaptive_paper_fees.py:936-944 (assertIsInstance(raised, SafetyError) gets None) fail at 4 of 7 shifted instants under load on this PR's original tree, on 791eab026 too, and at the real clock with no harness on both trees; run alone it passed 6 of 6 per tree. The class does not use FixtureClock and is untouched. In the builder's runs on the first base fees was 62/62 at all 16 instants, and four runs on the rebased commit (03:13Z to 03:15Z, load 28 or more) all passed 62/62. It is load-sensitive and pre-existing; R3 below names the open question.

Independent read of the original commit (Opus, read-only: verdict PASS, no blocker)

Read at 3d7e3635 (parent 791eab026), before the rebases; the rebased commit is checked against it by script (see Integration notes). The read reproduced the 10 failures on main at OVERNIGHT (TZ=UTC and TZ=America/New_York), the 7-instant boundary (main fails exactly in OVERNIGHT; the fix is green at all 7 for mover_native and admission), the red/green of the three fail-first tests with only the three wiring lines reverted (3 red, 1 green, at the real clock and in RTH), and a leak probe in one process (no fixture clock or default outlived its window). Follow-ups, none of which blocks and none of which this PR changes:

1. AdaptiveLaneFees.test_full_next_trial_budget_prevents_the_http_read (tests/test_adaptive_paper_fees.py:936-944) fails under load on both trees (the reader saw it at 4 of 7 instants; alone it passed 6 of 6 per tree). The class does not use FixtureClock. The commit-message figure "fees 62/62 at all 16 instants" is the first-base runs' result.
2. Two product wall-clock reads stay real under FixtureClock: runner.py:474 (observed_at) and transport.py:1242 (history_start). A fixture-clock receipt can carry an observed_at about 4.5 days after its own trial stamps. No assertion reads them today.
3. Seam: the commit patches the __kwdefaults__ of runner.Controller.__init__ and mover_strategy.MoverStrategy.__init__ from tests (no product line changes), which is a third option beside "a seam the code already has" and "a minimal parameter". native_strategy.AdaptiveStrategy.__init__ (native_strategy.py:26-27, default clock=time.time) is not patched and is imported lazily; the mover paths never import it under the patch, but a future mover-path test that first-imports it inside patched() would bind FixtureClock.time as its default for the rest of the process. One line in patched() (import it first, or patch its default too) closes that.
4. tests/adaptive_paper_hermetic.py's comment "later than every instant the fixtures embed" holds for every fixture the three modules read; tests/fixtures/adaptive_paper_citations/sources.json carries 2026-10-10 and 2026-10-11 but only tests/test_adaptive_paper_overnight.py reads it.
5. The positive test does not exercise the overnight_status_stale, overnight_not_tradable or overnight_halted branches or the boats feed entry, and no end-to-end overnight trial exists on this path (native_adapter.py:767-784 refuses OVERNIGHT dispatch by design), so the unit test is the only mover-path coverage of the refusal.
6. patch.dict restores __kwdefaults__ by clear() then update() on the shared dict: a thread constructing a Controller in that window would see no clock default (test-only, not observed).
7. The docstring statement "NautilusTrader's own clock and time.monotonic stay real" is unmeasured at rc5 (its Python surface exposes only Clock.new_test); cite the rc5 live-clock source or measure it.

Research queue:

• R1, candidate mechanism for the "ledger says filled, receipt has no native fill" class (untested): under FixtureClock the simulated broker stamps accepts and fills with the patched time.time_ns (mover_simulation.py:119,141,194,212), so OrderAccepted and OrderFilled ts_event would sit about 4.5 days before ts_init and before every other NautilusTrader stamp, if the engine's Rust clock stays real as the docstring says. On main both are real. Experiment, to run when the real clock is in a non-overnight session: run the sub-penny and x2 tests 100 times per tree under equal load, then once more on this tree with the fixture start moved to the real clock (offset 0). If the class vanishes at offset 0 the inversion is in the causal path and becomes a P2 on hermetic.py. The counts above do not separate the trees.
• R2: load-sensitive mover tests on both trees (test_x3_trails_out_at_085_of_the_running_high failed once on main in the reader's runs, test_exhausted_exits_hand_off_to_recovery_soon_after_the_hard_flatten once on this PR's tree).
• R3: why the filled per-minute budget does not stop the read in follow-up 1 under load (safety.py request-budget windowing against the adaptive run_main's clock).

Integration notes

• Lane lane:trading.

• The commit was built on 791eab026, rebased onto c735e8b2e (clean) and then onto 3a8a8461e (after Bound paper sell quote age independently of entry freshness #1015 and Add a local Claude review worker for essential pull request heads #953). On each rebase manifests/evidence.json was rebuilt from main's copy by re-registering exactly:

  • tests/adaptive_paper_hermetic.py
  • tests/test_adaptive_paper_fees.py
  • tests/test_adaptive_paper_mover_native.py
  • blueprints/us-equities/adaptive-paper/source-hashes.json

  The result is byte-identical to the rebase's own merge. Checked by script against the original commit (3d7e3635, parent 791eab026): the same 5 changed paths; git patch-id --stable equal for the three test files and tests/adaptive_paper_hermetic.py; the evidence.json rows that differ from main are exactly the four registered paths, each equal to the sha256 and size of the file at this commit; source-hashes.json differs from main in exactly the three test rows, each equal to the file's sha256; the manifest diff is 7 removed and 7 added lines, all sha256 or bytes; the original and this commit differ in the mover_native and source-hashes.json rows only because main's own change to that test file (Bound paper sell quote age independently of entry freshness #1015) is now part of the base.

• On a later rebase, take main's evidence.json, re-register the same four files, and recompute the three test rows in source-hashes.json if main touched those files.

• Bound paper sell quote age independently of entry freshness #1015's new admission test uses run_paper and inherits the fix.

• CI does not run these tests, so a reviewer needs a local run with the pinned native runtime.

Cross-family gate (read at 9514a87fe)

• Claude: the independent Opus read of the original commit (above) is PASS; the rebased commit is verified against it by script. My gate record is nas-pr1029-gate-9514a87f.md (trading-cc-read success at 05:28:25Z).
• Astra: ACK at this head, 0 blockers (file 12,731 B, sha256 a59845ba…, returned 05:21:23Z by the co-op's watch). On her reconstructed merge base the real overnight session fails 7 of 28 mover-native, 2 of 19 admission and 2 of 61 fees tests; the head and the merge with main 8b1ef0c3 pass 31 of 31, 19 of 19 and 62 of 62; exit-chain is 14 of 14 on all three; the three fail-first tests fail with completed_no_signals when only the clock wiring is disabled in memory; validate.py and final_stack.py --check rc 0 on head and merge. Her four P3: (1) the disclosed lazy-import boundary (follow-up 3 above); (2) fixture and metadata timestamps use different clocks (follow-up 2 above); (3) 18 hostname-bearing lines in the changed files are inherited, none added, 0 username hits (permitted by the 10-11 ruling); (4) the ordinary asynchronous native commands time out in her sandbox (her complete native results use a disclosed 50 ms heartbeat harness), so the ordinary commands should be repeated in the co-op's jail.
• Codex: no review yet (draft); the lander marks the PR ready and Codex follows.

SOTA sources

• Mechanism, in this repository at 3a8a8461ee60bd499efcb78156ce4c3c052430c5:
  • blueprints/us-equities/adaptive-paper/runner.py:934-958 (Controller._check_overnight_status; default clock=time.time at :828)
  • blueprints/us-equities/adaptive-paper/sessions.py:126-172 (session_at)
• The OVERNIGHT window comes from Alpaca's 24/5 trading FAQ, which sessions.py cites (https://docs.alpaca.markets/us/docs/245-trading.md, updated 2026-07-07, fetched 2026-10-10): 20:00 to 04:00 ET, Sunday evening through Friday morning.
• XNYS trade dates and holidays come from exchange_calendars 4.13.2, gerrymanoim/exchange_calendars at dbe38b1f6887434bbdd1a7d2df6ff8f1742a048a, exchange_calendars/exchange_calendar_xnys.py.
• The test clock uses CPython 3.12's unittest.mock (patch.object, patch.dict; https://docs.python.org/3.12/library/unittest.mock.html) on the functions' keyword-only defaults (__kwdefaults__; https://docs.python.org/3.12/reference/datamodel.html).
• Regression tests:
  • fail-first: MoverNativeEndToEnd.test_an_extended_hours_trial_keeps_the_fixture_clock_after_20_et
  • fail-first: MoverPaperCommandWiring.test_the_paper_command_keeps_the_fixture_clock_after_20_et
  • fail-first: MoverLaneFees.test_a_mover_trial_keeps_the_fixture_clock_after_20_et
  • positive gate test: MoverNativeEndToEnd.test_the_mover_controller_admits_an_overnight_entry_only_with_an_eligible_asset_status

Scope

• What this PR changes: the mover native, paper and fees test fixtures run on a fixed clock instead of the wall clock. No product source changes.
• Base commit: 3a8a8461ee60bd499efcb78156ce4c3c052430c5
• Lane: lane:trading
• Owned paths touched: tests/adaptive_paper_hermetic.py, tests/test_adaptive_paper_mover_native.py, tests/test_adaptive_paper_fees.py and blueprints/us-equities/adaptive-paper/source-hashes.json (trading); manifests/evidence.json (shared hot file, re-registration only).

Evidence-class table

Claim	Evidence class	Command / receipt
The overnight gate is the cause (mechanism)	source_review	runner.py:932-956, sessions.py:126-172
The boundary is the ET OVERNIGHT session, not the UTC date	local_integration	16-instant offset-clock runs of the three modules on main
The fail-first tests are red on main's runners and green with the fix	local_integration	runs at the real clock and at RTH, CLOSED and OVERNIGHT instants
The changed modules pass at any wall-clock time tested	local_integration	16 instants, the real clock, TZ=America/New_York, and fixture-to-engine offsets up to 732 days
The x2 and fees-budget intermittents predate this PR	local_integration	40 + 40 interleaved runs per tree (builder), 24 + 24 on the rebased commit, an independent reader's 30 + 30 and 7-instant runs
The tests themselves	synthetic	SYN fixtures; no broker, network or credential

Local commands run

$ python3 -m unittest tests.test_adaptive_paper_mover_native tests.test_alpaca_admission_regressions tests.test_adaptive_paper_fees tests.test_mover_exit_chain   # pinned native runtime, network-less jail, TZ=UTC
OK: 31, 18, 62 and 14 tests at the real clock (02:14Z) and at the measured instants
$ python3 -m unittest tests.test_adaptive_paper_source_hashes_manifest tests.test_evidence_manifest tests.test_trading_gates_bindings
OK: 25 tests
$ python3 scripts/validate.py
exit 0, "status": "passed"
$ python3 scripts/final_stack.py --check
exit 0

Decision record

None: a test-only change that alters no rule, doctrine or trading behaviour.

Host evidence

Not applicable: no files under evidence/hosts/ change.

Checklist

• New/changed GitHub Actions are pinned to a full commit SHA with a version comment (none changed).
• New/changed workflows declare top-level permissions: {} and grant each job only what it needs (none changed).
• No secrets are printed, logged or committed; no new required secret was added.
• No new paid hosting, subscription or billing surface was introduced.
• Peer-owned untracked files and worktrees were preserved.

🤖 Generated with Claude Code