Repository navigation
Data lake source layer: governed backfill, scope fingerprint, page-level capture, stock_bars_1min with a disk-gated pilot - #208
Merged
Conversation
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
… (news archive 2015-2026 backfilled) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ot disk gate Extends blueprints/us-equities/data-lake/backfill.py per the 2026-09-24 convergence record (D1-D2, backlog items 7-8) and critique items X2-X4: - Scope header: <dataset>/scope.json records the code sha256, endpoint, fixed params, symbol task-list sha256 and span. A run with a different scope, an edited or unreadable header, or a header-less manifest is refused (exit 2) before any request; a lock file admits one run per dataset. - Page-level raw capture for new datasets: each page, requested with Accept-Encoding: gzip, is stored as received as it arrives, with its request params, sha256, decoded sha256 and sizes in the task's manifest row. Bars outside 04:00-20:00 ET are quarantined, not failed. news keeps its record capture (documented; the 2015-2026 archive has no header and is not resumed). - stock_bars_1min: SIP 1Min raw bars with extended hours, 100 symbols x one month per task, drawn from the asof-aware monthly universe (`backfill.py universe` over the identity-deduplicated broad-universe daily dataset; asof is its naming date, 2026-09-21), at most 6,000 requests/min. - Pilot: --pilot N reports raw bytes per page, bytes per bar and peak RSS, and projects full-span disk use, refusing above 40% of free disk. A full run requires a passing pilot of the same scope. - verify re-hashes a dataset against its manifest. Evidence: - Synthetic fixtures: 25 tests, no network; 11 of 11 mutations killed. - Local integration (read-only): the universe build on the retained daily dataset reproduces 17,997 symbols, 24,195,741 symbol-sessions, 11,753 tasks and the 25/47/167 last-bar delistings. `verify news` re-hashed 4,284 of 4,284 day files with 0 problems. - No provider request was made. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…tals Independent review of caad69c2 against the 2026-09-24 convergence record (D1-D2) and critique items X2-X4: - Every stock_bars_1min manifest row now carries the dataset's scope fingerprint (D1: "give each task a scope fingerprint"). Resume and the pilot report count only rows of the header's scope, and verify reports a missing or edited header and any row naming another fingerprint. - --pilot is bounded to 20 tasks. The pilot runs before any disk gate; a synthetic probe showed --pilot 1000000 fetching all 6 planned tasks and only then reporting "refused". - A header whose scope is not an object is refused instead of raising. - README: the news archive holds 4,284 days, 2,217,343 records and 46,650 pages from two runs; the earlier figures were the second run's alone. The disk gate's missing runtime floor is listed as a limit. - The rate-cap test now isolates the cap with a 6,000/min control. Evidence: - Synthetic fixtures: 28 tests, 0 skipped, no network (system python3 and the pinned engine venv, both 3.12.3); 20 of 20 mutations killed. - Local integration (read-only): the universe rebuild is byte-identical (sha256 a7c98f5c...); planning 2016-01-01..2026-08-31 gives 11,620 tasks and 24,024,163 symbol-sessions; verify news re-hashed 4,284 of 4,284 day files with 0 problems. - scripts/validate.py passed. - No provider request was made. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…odes, plan-aware verify) Resolves both reviews of 5823e7d1 (convergence record 2026-09-24, D1-D2, items 7-8; critique X2-X4): - Credentials stay on the fixed endpoint: the client's opener refuses every redirect before a follow-up request is built and ignores environment proxies. - Durability: page, day, scope.json and pilot.json writes are fsynced with their directory, and each manifest append is fsynced. A task (or news day) counts as done only while its recorded files exist at their recorded sizes, so a crash that kept a row but not its files leads to a refetch. - Licensed data never lands in a checkout: an --out (or universe --out) at or below any .git file or directory, symlinks resolved, is refused (exit 2) before the credentials are read. - A run is never reported as done when it is not. A drained run exits 3, a leftover STOP is refused at start, and the scope records the planned task count and the sha256 of the task ids. verify reports planned tasks with no row (complete: false, exit 1), tasks outside the plan, and a missing manifest or malformed rows, instead of raising. - One backfill per host: a host-wide flock lease covers every dataset and --out, within the shared data limit. - Ctrl-C or a failed manifest append halts the client and cancels queued work. Submission is bounded to --workers and rows are released once written, and the quarantine list is capped at 100 with an exact count. - Pilot: wire bytes and pages by Content-Encoding are reported. The D2 call-estimate flag gives an "overturn" verdict, and a full run needs --accept-pilot-flags REASON after it. The verdict weighs the remaining bytes, and a refused full run writes no header. - Disk gates keep a 20 GiB floor plus a normalized-layer reserve (16.8 bytes/bar, D2 proposal) free. They re-check before each task and drain when that no longer holds. - news: new pulls end each day at 23:59:59.999999999Z, since both bounds are inclusive. The README records the archive's midnight overlap (2,217,343 records, 2,217,341 distinct ids) and that consumers dedupe by id. Evidence: - Synthetic fixtures (no network): 67 tests. Under the pinned engine venv 0 are skipped (DuckDB 1.5.5); system python3 skips the DuckDB test. Against the 5823e7d1 backfill.py, 44 of the 67 fail; the rest cover behaviour that already held. 66 of 67 targeted mutants are killed. The survivor removes cancel_futures, which the bounded window plus the halt check make equivalent. - Local integration (read-only): verify news on the retained archive gives 4,284 of 4,284 files and 0 problems. An id scan finds 2,217,343 records and 2,217,341 distinct ids. From the retained universe: 11,620 tasks, 24,024,163 symbol-sessions, the pilot picks, and D2's 425,763-call estimate. - scripts/validate.py passed. - No provider request was made, and the gzip wire encoding, page fullness and the nanosecond end bounds are still unexercised natively. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
seathatflowsinourveins
force-pushed
the
claude/data-lake-20260924
branch
from
September 24, 2026 23:37
1d52ed2 to
86d9592
Compare
seathatflowsinourveins
changed the title
Owner
Author
|
CI note (2026-09-25). Nothing in the PR needs to change. The next push, or a close and reopen, runs the current 🤖 Generated with Claude Code |
seathatflowsinourveins
added
the
lane:trading
9 of 22 tasks
Brings the branch onto main 5b6014e so CI, including the required validate-macos check, runs on the current workflows. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Owner
Author
|
Trading lane, 2026-09-25: carrying this PR to merge. Under the agreed lane split this PR is trading-lane work, and it has had no commits since 2026-09-24. The trading-lane coordinator is carrying it, per the user's approval of dormant-owner merges. The original owner can still take it back by replying here. What changed:
Checks on the merged tree:
Scope: this merge adds the source layer only. No minute-bar backfill has run; the disk-gated pilot comes first, as the PR states. The retained I'll merge once the required checks pass. 🤖 Generated with Claude Code |
seathatflowsinourveins
added a commit
that referenced
this pull request
Sep 25, 2026
… incentive board and data lake, and the recounted WSL host (#288) - simulation: 866 to 884 fills per min at 70 ms exact release, 30 of 30 minutes, and 672 at 1 s (#254, sim-capacity/receipts/20260925-elite-tier.json). - paper: adds the margin financing model (#243); still blocked on the paper keys (HTTP 401). - mover-early-entry-studies: v1 to v3, with the v3 core merged (#190) and not frozen. - New gates: - incentive-board-forward (#187, #220): 0 orders; - data-lake-source (#208): minute-bar pilot not run. - wsl-workstation-host: the foundation lane's wording. Counts were recounted on main 0074a0c by the last independent_session verdict per receipt, superseded excluded: 69 receipts, 56 agree, 1 needs changes, 0 awaiting review, 12 superseded (#285). The foundation lane acked the strings and the recount rule. Co-authored-by: Scout <scout@local> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
blueprints/us-equities/data-lake/backfill.pyis the lake's governed source layer for Alpaca historical REST:Additions from the 2026-09-24 data-and-execution convergence record (items 7–8, critic X2–X4):
newskeeps its documented record-level day files.stock_bars_1min: SIP 1-minute bars with raw adjustment, from an asof-aware universe that includes delisted names. Each task covers 100 symbols × one month, extended hours included.Native result already retained
newsfor 2015-01-01..2026-09-23:No minute-bar backfill has run yet; the pilot comes first.
Review trail
The build workflow ran two independent reviews: 8 majors in total, all fixed in a25f5d04. The recheck approved, with 3 minors left.
Checks
tests.test_data_lake_backfill: 67 OK (1 DuckDB test skipped on system python; 0 skipped in the pinned venv). Synthetic openers only.python3 -m unittest: 4,403 OK (440 skipped).scripts/validate.py: passed.🤖 Generated with Claude Code