Build initial risk/data management functionality for statistical arbitrage

[forstmeier]

Overview

Changes

• build data fetching logic
• update Alpaca client filtering for shortable assets
• add initial risk management/statistical arbitrage functions
• expand unit test suite
• update server logic
• include basic stubs for followup work
• consolidate Expand portfolio manager rebalancing logic #786 into this pull request
• consolidate Build initial beta/regime risk management features #787 into this pull request

Context

I'm separating this into chunks just because the pull request will be pretty big regardless and I want to be able to apply as many of the bot feedback as I can.

These are the data resources and scripts I used to test it - they lived in tools/ and tools/data/ but I don't think they need to be checked into the repository. I liked what they generated though and it showed everything working.

stat_arb_test.py
build_stat_arb_test_data.py
alpaca_shortable_cache.json
stat_arb_test_prices.csv
stat_arb_test_sectors.csv

Summary by CodeRabbit

• New Features

  • Ensemble prediction blending, statistical‑arbitrage pair selection, volatility‑parity position sizing, shortable‑ticker filtering, historical price & equity ingestion, market beta/regime analytics, and textual portfolio reports.

• Improvements

  • Stronger portfolio/pair validation, clearer error handling, adjusted portfolio empty-response behavior, refined open/close position flow, and added pair identifiers throughout.

• Tests

  • Extensive unit tests covering ingestion, consolidation, pair selection, sizing, beta/regime, reporting, and broker interactions.

• Dependencies

  • Added scipy (>=1.17.1).

[coderabbitai]

Caution

Review failed

Failed to post review comments

📝 Walkthrough

Walkthrough

Adds a pairs-first statistical-arbitrage pipeline: data fetchers, model consolidation, pair selection, volatility- and beta-based sizing, regime/beta analytics, Alpaca shortability checks, portfolio schema changes (pair_id), server orchestration updates, new exceptions, and extensive unit tests and Rust datamanager adjustments.

Changes

Cohort / File(s)	Summary
Dependency applications/portfoliomanager/pyproject.toml	Added runtime dependency scipy>=1.17.1.
Alpaca client applications/portfoliomanager/src/portfoliomanager/alpaca_client.py	Added get_shortable_tickers(...) and asset-related imports to query active US equities and filter shortable/easy_to_borrow tickers.
Data fetchers applications/portfoliomanager/src/portfoliomanager/data_client.py	New HTTP fetch utilities: fetch_historical_prices, fetch_equity_details, fetch_spy_prices; return Polars frames and raise PriceDataUnavailableError on fetch failures.
Consolidation applications/portfoliomanager/src/portfoliomanager/consolidation.py	New compute_ticker_volatility and consolidate_predictions to blend model quantiles into ensemble alpha/confidence, normalize, and enrich with volatility and sector.
Statistical arbitrage applications/portfoliomanager/src/portfoliomanager/statistical_arbitrage.py	New module: build price matrix, compute spread z-score/hedge, candidate construction, correlation filtering, greedy non-overlap selection via select_pairs.
Risk management applications/portfoliomanager/src/portfoliomanager/risk_management.py	Replaced prior portfolio builders with size_pairs_with_volatility_parity and _apply_beta_neutral_weights; added REQUIRED_PAIRS and related constants; uses InsufficientPairsError.
Beta & Regime applications/portfoliomanager/src/portfoliomanager/beta.py, .../regime.py	New compute_market_betas and compute_portfolio_beta; new classify_regime and RegimeResult typed dict for regime detection.
Server orchestration applications/portfoliomanager/src/portfoliomanager/server.py	Major flow refactor: async lifespan, fetch external data, get_raw_predictions, consolidate signals, filter shortable tickers, select pairs, compute betas/regime, size pairs, updated function signatures and error handling.
Schema & validation applications/portfoliomanager/src/portfoliomanager/portfolio_schema.py	Added pair_id field to portfolio schema, pairs_schema, and check_pair_tickers_different validator enforcing long/short differ.
Reporting applications/portfoliomanager/src/portfoliomanager/report.py	New textual report formatters for regime, betas, consolidation, pairs, and portfolio summaries.
Exceptions applications/portfoliomanager/src/portfoliomanager/exceptions.py	Removed InsufficientPredictionsError; added PriceDataUnavailableError and InsufficientPairsError.
Tests (Python) applications/portfoliomanager/tests/*	Added/updated extensive tests for alpaca client, consolidation, data client, portfolio schema, risk sizing, stat-arb, beta, regime, server flows, report formatting, and prior-portfolio handling.
Datamanager (Rust) applications/datamanager/src/...	Added pair_id: String to Portfolio struct, propagated through storage/query/result mapping and tests; portfolio GET now returns empty JSON array (200) for first-run / empty-file cases.
Datamanager tests & handlers applications/datamanager/tests/*, applications/datamanager/src/portfolios.rs, applications/datamanager/src/storage.rs	Updated tests and handlers to include pair_id in payloads/fixtures; unified query execution and projection to include pair_id; adjusted empty-file GET behavior.
Tooling config pyproject.toml	Added Ruff configuration blocks (linting rules).

Sequence Diagram(s)

sequenceDiagram
    participant Server as Server
    participant DM as DataManager API
    participant Models as Prediction Service
    participant Consolidation as Consolidation
    participant Alpaca as Alpaca API
    participant Pairs as Pair Selector
    participant Risk as Risk Manager

    Server->>DM: fetch_historical_prices(reference_date, lookback_days)
    DM-->>Server: historical_prices
    Server->>DM: fetch_equity_details()
    DM-->>Server: equity_details

    Server->>Models: get_raw_predictions()
    Models-->>Server: model_predictions

    Server->>Consolidation: consolidate_predictions(model_predictions, historical_prices, equity_details)
    Consolidation-->>Server: consolidated_signals

    Server->>Alpaca: get_shortable_tickers(consolidated_signals.tickers)
    Alpaca-->>Server: shortable_tickers

    Server->>Pairs: select_pairs(consolidated_signals.filtered, historical_prices)
    Pairs-->>Server: candidate_pairs

    Server->>Risk: size_pairs_with_volatility_parity(candidate_pairs, maximum_capital, timestamp, market_betas, exposure_scale)
    Risk-->>Server: sized_positions

    Server->>Alpaca: open/close positions (orders)
    Alpaca-->>Server: order confirmations

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

• portfolio fixes #669: Overlapping refactor touching alpaca_client, risk_management, and server — likely parallel work on portfolio flow and pair handling.
• Refactor datamanager module #635: Related datamanager changes modifying Portfolio/data structures and storage/query logic; directly relevant to added pair_id.
• Build initial risk management resources #625: Changes to risk management and portfolio schema that overlap with the new sizing and schema validation work.

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 13.55% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The pull request title clearly and concisely summarizes the main change: building initial risk/data management functionality for statistical arbitrage. It directly reflects the primary objective and scope of the changeset.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

• 📝 Generate docstrings (stacked PR)
• 📝 Generate docstrings (commit on current branch)

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch statistical-arbitrage-phase-one

Tip

Try Coding Plans. Let us write the prompt for your AI agent so you can ship faster (with fewer bugs).
Share your feedback on Discord.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[greptile-apps]

Greptile Summary

This PR builds the initial statistical arbitrage pipeline for the portfolio manager, adding pair selection, volatility-parity sizing, beta-neutral optimization, regime classification, and hold/stop-loss logic. Most previously flagged issues (zero-price filter, isinf hedge ratio guard, pairs_schema validation, pair_id in position rows, lookback window increase) have been addressed.

Two issues remain:

• Logic bug in consolidation.py line 63: predictions_df.sort("timestamp").group_by("ticker").last() does not guarantee the most-recent row per ticker because Polars' group_by is hash-based and does not preserve sort order within groups. This can silently mix stale predictions with current ones. The same hazard was correctly fixed in compute_ticker_volatility using sort_by inside the aggregation — apply the same fix here.

• Unguarded .row(0) in report.py lines 145–146: Calling .row(0, named=True) on potentially empty filtered DataFrames will raise OutOfBoundsError. Although the production pipeline ensures both legs exist for every pair, this function should guard against empty results before accessing rows.

Confidence Score: 3/5

• Contains one logic bug in signal consolidation that can silently produce stale alpha signals; a secondary defensive programming gap in the report formatter.
• The PR implements stat-arb core logic well and passes comprehensive tests. However, the logic bug in consolidate_predictions (line 63) is significant: when a model returns multiple predictions per ticker at different timestamps, group_by().last() may return an arbitrary row instead of the most-recent one, silently corrupting ensemble signal blending. This is a production correctness issue. The report.py gap is lower priority since the function is not yet in the live pipeline, but both warrant fixes before merge.
• applications/portfoliomanager/src/portfoliomanager/consolidation.py (must fix before merge); applications/portfoliomanager/src/portfoliomanager/report.py (should fix before use).

Comments Outside Diff (2)

1. applications/portfoliomanager/src/portfoliomanager/consolidation.py, line 63 (link)

   group_by().last() does not guarantee the most-recent row per ticker because Polars' group_by is hash-based and does not preserve sort order within groups. The .sort("timestamp") on line 63 orders the whole frame, but once rows enter the grouper their relative order is undefined.

   This is the same ordering hazard that was correctly fixed in compute_ticker_volatility (lines 14–17), which uses sort_by inside the aggregation expression to guarantee per-group ordering. The same fix should be applied here:

   latest_predictions = (
       predictions_df
       .group_by("ticker")
       .agg(pl.all().sort_by("timestamp").last())
   )

2. applications/portfoliomanager/src/portfoliomanager/report.py, line 145-146 (link)

   .row(0, named=True) is called on lines 145–146 without checking whether the filtered DataFrames are empty. If no LONG or SHORT row exists for a given pair_id, this will raise polars.exceptions.OutOfBoundsError with a confusing error message.

   While the production pipeline ensures both legs are present for every selected pair, this function would crash during debugging or if called with inconsistent arguments. Adding a guard makes the failure explicit:

   long_rows = pair_rows.filter(pl.col("side") == "LONG")
   short_rows = pair_rows.filter(pl.col("side") == "SHORT")
   if long_rows.is_empty() or short_rows.is_empty():
       lines.append(f"  {pair_id:<{_W_PAIR_ID + 2}} (incomplete pair — missing leg)")
       continue
   long_row = long_rows.row(0, named=True)
   short_row = short_rows.row(0, named=True)

_{Last reviewed commit: a93674f}

[Copilot]

Pull request overview

This PR refactors the portfolio manager from a prediction-based long/short strategy to a statistical arbitrage (pairs trading) approach. It introduces data fetching utilities, a consolidation pipeline for blending model predictions, pair selection via cointegration analysis, and volatility-parity-based position sizing. The old add_predictions_zscore_ranked_columns / create_optimal_portfolio approach is replaced end-to-end.

Changes:

• Added statistical_arbitrage.py for cointegration-based pair selection, consolidation.py for blending model predictions with volatility and sector data, and data_client.py for fetching historical prices and equity details from the data manager.
• Rewrote risk_management.py to size pairs using inverse-volatility weighting instead of equal-dollar allocation across ranked predictions, and updated server.py with the new pipeline (data fetch → consolidation → shortability filter → pair selection → volatility-parity sizing).
• Expanded the test suite with comprehensive tests for all new modules and updated existing tests, added pairs_schema validation to portfolio_schema.py, and introduced new custom exceptions (PriceDataUnavailableError, InsufficientPairsError).

Reviewed changes

Copilot reviewed 15 out of 17 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
statistical_arbitrage.py	New module for cointegrated pair selection using correlation filtering, spread z-scores, and greedy pair assignment
consolidation.py	New module for blending multi-model predictions with realized volatility and sector data
data_client.py	New module for fetching historical prices (parquet) and equity details (CSV) from the data manager
risk_management.py	Replaced equal-allocation portfolio construction with inverse-volatility-parity pair sizing
server.py	Updated portfolio creation pipeline with new data fetching, consolidation, shortability filtering, pair selection, and sizing steps
portfolio_schema.py	Added pairs_schema with check_pair_tickers_different validation and optional pair_id column to portfolio_schema
exceptions.py	Added PriceDataUnavailableError and InsufficientPairsError; removed InsufficientPredictionsError
alpaca_client.py	Added get_shortable_tickers method for filtering assets by shortability and easy-to-borrow status
pyproject.toml	Added scipy>=1.17.1 dependency
test_statistical_arbitrage.py	New tests for price matrix building, spread z-score computation, and pair selection edge cases
test_consolidation.py	New tests for prediction consolidation, blending, and error handling
test_data_client.py	New tests for data fetching with mocked HTTP responses
test_risk_management.py	Rewrote tests for the new volatility-parity sizing function
test_alpaca_client.py	New tests for get_shortable_tickers and full coverage of AlpacaClient methods
test_portfolio_schema.py	Added tests for new schema checks and pairs_schema validation
.flox/env/manifest.lock	Trailing newline fix

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[chrisaddy]

Review comments on initial statistical arbitrage build.

[Copilot]

Pull request overview

Copilot reviewed 15 out of 17 changed files in this pull request and generated no new comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 29 out of 31 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

applications/datamanager/src/data.rs:151

• The pair_id column is not normalized to uppercase, unlike ticker, side, and action which all have .str().to_uppercase() applied (lines 148-150). The test sample_portfolio_lowercase() uses pair_id: "aapl-googl" but test_create_portfolio_dataframe_uppercase_normalization doesn't verify pair_id is uppercased. Since pair_id is derived from tickers (e.g. "AAPL-MSFT"), this could cause mismatches when querying by pair_id if lowercase data is ever written. Consider adding .with_columns([col("pair_id").str().to_uppercase().alias("pair_id")]) alongside the other normalization steps.

    debug!("Normalizing ticker, side, and action columns to uppercase");
    let portfolio_dataframe = portfolio_dataframe
        .lazy()
        .with_columns([col("ticker").str().to_uppercase().alias("ticker")])
        .with_columns([col("side").str().to_uppercase().alias("side")])
        .with_columns([col("action").str().to_uppercase().alias("action")])
        .collect()?;

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.