Skip to content

Build initial prediction engine service #559

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
chrisaddy merged 12 commits into from
May 31, 2025
Merged

Build initial prediction engine service #559

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
3b76b59
Build initial prediction engine service
forstmeier May 26, 2025
ff4ad48
Update application/predictionengine/src/predictionengine/loss_functio…
chrisaddy May 31, 2025
8d83bf9
Update application/predictionengine/src/predictionengine/main.py
chrisaddy May 31, 2025
4ca456e
Update application/predictionengine/src/predictionengine/main.py
chrisaddy May 31, 2025
48fac02
Update application/predictionengine/src/predictionengine/main.py
chrisaddy May 31, 2025
e9f6982
Update application/predictionengine/src/predictionengine/post_process…
chrisaddy May 31, 2025
e112e39
Update application/predictionengine/src/predictionengine/multi_head_s…
chrisaddy May 31, 2025
62f443b
Update application/predictionengine/src/predictionengine/miniature_te…
chrisaddy May 31, 2025
1dc1c4e
Merge branch 'master' into 05-25-build_initial_prediction_engine_service
chrisaddy May 31, 2025
dc025bc
fix linting
chrisaddy May 31, 2025
502f22a
fix linting introduced
chrisaddy May 31, 2025
1c3ed7e
update linting/types
chrisaddy May 31, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .dockerignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
.venv/
**/__pycache__/
2 changes: 1 addition & 1 deletion .github/workflows/test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,4 +19,4 @@ jobs:
uses: coverallsapp/github-action@v2
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
path-to-lcov: .python_coverage.xml
path-to-lcov: coverage_output/.python_coverage.xml
4 changes: 3 additions & 1 deletion .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,6 @@ infrastructure/Pulumi.infrastructure.yaml
.envrc
.coverage*
.coverage/
coverage.xml
coverage.xmlcoverage.xml
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot May 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Fix the duplicated coverage.xml entry.

The line appears to have a concatenation error with coverage.xmlcoverage.xml. This should be just coverage.xml.

Apply this diff to fix the typo:

-coverage.xmlcoverage.xml
+coverage.xml
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
coverage.xmlcoverage.xml
coverage.xml
🤖 Prompt for AI Agents 
In the .gitignore file at line 12, there is a duplicated entry
"coverage.xmlcoverage.xml" due to a concatenation error. Replace this line with
a single "coverage.xml" entry to correctly ignore the coverage.xml file.


todos.md
11 changes: 7 additions & 4 deletions .mise.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,11 @@ uvx ty check
[tasks."python:test"]
description = "Run Python tests"
run = """
mkdir -p coverage_output
docker compose down --volumes --remove-orphans
docker compose build tests
docker compose run -T tests
docker compose run --rm --no-TTY tests
docker compose down --volumes --remove-orphans
"""

[tasks."application:service:run:production"]
Expand All @@ -56,21 +59,21 @@ uv run uvicorn src.{{arg(name="service_name")}}.main:application --reload
description = "Run integration tests"
run = """
cd application/{{arg(name="service_name")}}
docker-compose up --build --abort-on-container-exit --remove-orphans
docker compose up --build --abort-on-container-exit --remove-orphans
"""

[tasks."application:service:test:behavioral"]
description = "Run behavioral tests"
run = """
cd application/{{arg(name="service_name")}}
docker-compose up --build --abort-on-container-exit
docker compose up --build --abort-on-container-exit
"""

[tasks."application:service:cleanup:behavioral"]
description = "Clean up behavioral tests"
run = """
cd application/{{arg(name="service_name")}}
docker-compose down -v
docker compose down -v
"""

[tasks."lint"]
Expand Down
28 changes: 26 additions & 2 deletions Dockerfile.test
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,33 @@
FROM python:3.13

COPY --from=ghcr.io/astral-sh/uv:latest /uv /bin/uv

RUN apt-get update && apt-get install -y \
build-essential \
clang \
libc6-dev \
linux-libc-dev \
&& rm -rf /var/lib/apt/lists/*

ENV PYTEST_ADDOPTS="--rootdir=/tests"
ENV PYTHON=1

WORKDIR /tests

COPY pyproject.toml uv.lock .
RUN uv sync --all-groups
COPY pyproject.toml uv.lock ./

COPY application/datamanager/pyproject.toml ./application/datamanager/

COPY application/positionmanager/pyproject.toml ./application/positionmanager/

COPY application/predictionengine/pyproject.toml ./application/predictionengine/

COPY infrastructure/pyproject.toml ./infrastructure/

COPY workflows/pyproject.toml ./workflows/

RUN uv sync --all-packages --dev

COPY application/ ./application/

RUN mkdir -p /tests/coverage_output
4 changes: 3 additions & 1 deletion application/datamanager/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
FROM python:3.13
FROM python:3.12

COPY --from=ghcr.io/astral-sh/uv:latest /uv /bin/uv

WORKDIR /app

COPY pyproject.toml ./

RUN uv sync --no-dev

COPY ./src ./src
Expand Down
9 changes: 2 additions & 7 deletions application/datamanager/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,8 @@ dependencies = [
"loguru>=0.7.3",
"google-cloud-storage>=2.16.0",
"httpx>=0.28.1",
"datamanager"
"prometheus-fastapi-instrumentator>=7.1.0",
"loguru>=0.7.3",
]

[tool.hatch.build.targets.wheel]
Expand All @@ -21,9 +22,3 @@ packages = ["datamanager"]
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[dependency-groups]
dev = [
"behave>=1.2.6",
"requests>=2.31.0",
]
6 changes: 3 additions & 3 deletions application/datamanager/src/datamanager/config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,14 @@


class Polygon(BaseModel):
api_key: str | None = Field(default=os.getenv("POLYGON_API_KEY"))
api_key: str = Field(default=os.getenv("POLYGON_API_KEY", ""))
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot May 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Consider validation for required configuration fields.

While defaulting to empty strings provides type consistency, it may mask configuration errors. Consider adding validation to ensure critical fields like API keys and project identifiers are properly configured.

Also note the inconsistency: credentials_path on line 24 still uses the old pattern without an empty string default.

 class GCP(BaseModel):
     bucket: Bucket = Bucket()
-    credentials_path: str = os.getenv("GOOGLE_APPLICATION_CREDENTIALS", "")
+    credentials_path: str = Field(default=os.getenv("GOOGLE_APPLICATION_CREDENTIALS", ""))

Consider adding validation to ensure required fields are non-empty:

from pydantic import BaseModel, Field, field_validator

class Polygon(BaseModel):
    api_key: str = Field(default=os.getenv("POLYGON_API_KEY", ""))
    
    @field_validator('api_key')
    @classmethod
    def validate_api_key(cls, v):
        if not v:
            raise ValueError('POLYGON_API_KEY environment variable is required')
        return v

Also applies to: 14-15

🤖 Prompt for AI Agents 
In application/datamanager/src/datamanager/config.py at line 8 and also lines
14-15 and 24, the current use of default empty strings for critical config
fields like api_key and credentials_path can hide missing configuration errors.
To fix this, add pydantic field validators for these fields that check if the
value is non-empty and raise a ValueError if not. This ensures required
environment variables are properly set and prevents silent misconfiguration.

base_url: str = "https://api.polygon.io"
daily_bars: str = "/v2/aggs/grouped/locale/us/market/stocks/"


class Bucket(BaseModel):
name: str | None = Field(default=os.getenv("DATA_BUCKET"))
project: str | None = Field(default=os.getenv("GCP_PROJECT"))
name: str = Field(default=os.getenv("DATA_BUCKET", ""))
project: str = Field(default=os.getenv("GCP_PROJECT", ""))

@computed_field
def daily_bars_path(self) -> str:
Expand Down
4 changes: 3 additions & 1 deletion application/positionmanager/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,13 @@
FROM python:3.13
FROM python:3.12

COPY --from=ghcr.io/astral-sh/uv:latest /uv /bin/uv

ENV PYTHONPATH=/app/src

WORKDIR /app

COPY pyproject.toml ./

RUN uv sync --no-dev

COPY ./src ./src
Expand Down
9 changes: 7 additions & 2 deletions application/positionmanager/src/positionmanager/clients.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,12 @@ def __init__(

def get_cash_balance(self) -> Money:
account = self.trading_client.get_account()
return Money.from_float(float(account.cash))
cash_balance = getattr(account, "cash", None)

if cash_balance is None:
raise ValueError("Cash balance is not available")

return Money.from_float(float(cash_balance))

def place_notional_order(
self,
Expand Down Expand Up @@ -89,7 +94,7 @@ def get_data(

data = (
data.sort("date")
.pivot(index="date", columns="ticker", values="close_price")
.pivot(on="ticker", index="date", values="close_price")
.with_columns(pl.all().exclude("date").cast(pl.Float64))
)

Expand Down
12 changes: 5 additions & 7 deletions application/positionmanager/src/positionmanager/main.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,9 @@
from datetime import datetime, timedelta
import polars as pl
from typing import Dict, Any

from .models import Money, DateRange, PredictionPayload
from .clients import AlpacaClient, DataClient
from .portfolio import PortfolioOptimizer

from prometheus_fastapi_instrumentator import Instrumentator


Expand All @@ -25,12 +23,12 @@ def get_health() -> dict[str, str]:
@application.post("/positions")
def create_position(payload: PredictionPayload) -> Dict[str, Any]:
alpaca_client = AlpacaClient(
api_key=os.getenv("ALPACA_API_KEY"),
api_secret=os.getenv("ALPACA_API_SECRET"),
api_key=os.getenv("ALPACA_API_KEY", ""),
api_secret=os.getenv("ALPACA_API_SECRET", ""),
Comment on lines +26 to +27
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot May 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Avoid masking configuration errors with empty string defaults.

Providing empty string defaults for critical configuration like API keys and service URLs masks configuration errors instead of failing fast. The DataClient.get_data method explicitly checks for empty datamanager_base_url and raises a ValueError, and empty API keys will cause authentication failures at runtime.

Consider these better approaches:

Option 1: Validate at startup

+def validate_config():
+    required_vars = ["ALPACA_API_KEY", "ALPACA_API_SECRET", "DATAMANAGER_BASE_URL"]
+    missing = [var for var in required_vars if not os.getenv(var)]
+    if missing:
+        raise ValueError(f"Missing required environment variables: {missing}")
+
+@application.on_event("startup")
+async def startup_event():
+    validate_config()

Option 2: Keep None and handle explicitly

-        api_key=os.getenv("ALPACA_API_KEY", ""),
-        api_secret=os.getenv("ALPACA_API_SECRET", ""),
+        api_key=os.getenv("ALPACA_API_KEY") or raise ValueError("ALPACA_API_KEY not set"),
+        api_secret=os.getenv("ALPACA_API_SECRET") or raise ValueError("ALPACA_API_SECRET not set"),

Also applies to: 31-31

🤖 Prompt for AI Agents 
In application/positionmanager/src/positionmanager/main.py around lines 26 to 27
and line 31, remove the empty string defaults from the os.getenv calls for
critical configuration values like API keys and service URLs. Instead, allow
these to be None if not set, and add explicit validation at startup to check for
missing environment variables, raising an error immediately if any are absent.
This prevents masking configuration errors and ensures the application fails
fast with clear error messages.

paper=os.getenv("ALPACA_PAPER", "true").lower() == "true",
)

data_client = DataClient(datamanager_base_url=os.getenv("DATAMANAGER_BASE_URL"))
data_client = DataClient(datamanager_base_url=os.getenv("DATAMANAGER_BASE_URL", ""))

portfolio_optimizer = PortfolioOptimizer(
minimum_portfolio_tickers=int(os.getenv("MINIMUM_PORTFOLIO_TICKERS", "5")),
Expand Down Expand Up @@ -134,8 +132,8 @@ def create_position(payload: PredictionPayload) -> Dict[str, Any]:
@application.delete("/positions")
def delete_positions() -> Dict[str, Any]:
alpaca_client = AlpacaClient(
api_key=os.getenv("ALPACA_API_KEY"),
api_secret=os.getenv("ALPACA_API_SECRET"),
api_key=os.getenv("ALPACA_API_KEY", ""),
api_secret=os.getenv("ALPACA_API_SECRET", ""),
paper=os.getenv("ALPACA_PAPER", "true").lower() == "true",
)

Expand Down
19 changes: 19 additions & 0 deletions application/predictionengine/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
FROM python:3.12

COPY --from=ghcr.io/astral-sh/uv:latest /uv /bin/uv

ENV PYTHONPATH=/app/src

WORKDIR /app

COPY pyproject.toml ./

RUN uv sync --no-dev

COPY ./src ./src

COPY ./miniature_temporal_fusion_transformer.safetensor ./src/predictionengine/miniature_temporal_fusion_transformer.safetensor

EXPOSE 8080

ENTRYPOINT ["uv", "run", "uvicorn", "predictionengine.main:application", "--host", "0.0.0.0", "--port", "8080", "--app-dir", "src"]
20 changes: 20 additions & 0 deletions application/predictionengine/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
[project]
name = "predictionengine"
version = "0.1.0"
description = "Prediction engine service"
requires-python = ">=3.12" # possibly 3.10
dependencies = [
"fastapi>=0.115.12",
"uvicorn>=0.34.2",
"tinygrad>=0.10.3",
"polars>=1.29.0",
"category-encoders>=2.8.1",
"requests>=2.31.0",
]

[tool.hatch.build.targets.wheel]
packages = ["predictionengine"]

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
Loading