Skip to content
Merged
3 changes: 3 additions & 0 deletions docs/my-website/docs/proxy/config_settings.md
Original file line number Diff line number Diff line change
Expand Up @@ -934,6 +934,9 @@ router_settings:
| PROXY_BASE_URL | Base URL for proxy service
| PROXY_BATCH_WRITE_AT | Time in seconds to wait before batch writing spend logs to the database. Default is 10
| PROXY_BATCH_POLLING_INTERVAL | Time in seconds to wait before polling a batch, to check if it's completed. Default is 6000s (1 hour)
| PROXY_BATCH_POLLING_ENABLED | Set to `false` to disable the `CheckBatchCost` and `CheckResponsesCost` background polling jobs entirely. Useful for emergency mitigation on installs with large numbers of stale managed objects. Default is `true`
| MAX_OBJECTS_PER_POLL_CYCLE | Maximum number of managed objects (batches / responses) fetched per polling cycle. Prevents OOM on installs with many stale rows. Default is `50`
| MANAGED_OBJECT_STALENESS_CUTOFF_DAYS | Managed objects older than this many days in a non-terminal state are marked `stale_expired` at the start of each poll cycle and skipped. Default is `7`
| PROXY_BUDGET_RESCHEDULER_MAX_TIME | Maximum time in seconds to wait before checking database for budget resets. Default is 605
| PROXY_BUDGET_RESCHEDULER_MIN_TIME | Minimum time in seconds to wait before checking database for budget resets. Default is 597
| PYTHON_GC_THRESHOLD | GC thresholds ('gen0,gen1,gen2', e.g. '1000,50,50'); defaults to Python’s values.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,15 @@
Polls LiteLLM_ManagedObjectTable to check if the batch job is complete, and if the cost has been tracked.
"""

from litellm._uuid import uuid
from datetime import datetime
from datetime import datetime, timedelta, timezone
from typing import TYPE_CHECKING, Optional

from litellm._logging import verbose_proxy_logger
from litellm._uuid import uuid
from litellm.constants import (
MANAGED_OBJECT_STALENESS_CUTOFF_DAYS,
MAX_OBJECTS_PER_POLL_CYCLE,
)

if TYPE_CHECKING:
from litellm.proxy.utils import PrismaClient, ProxyLogging
Expand Down Expand Up @@ -49,6 +53,26 @@ async def _get_user_info(self, batch_id, user_id) -> dict:
verbose_proxy_logger.error(f"CheckBatchCost: could not look up user {user_id} for batch {batch_id}: {e}")
return {}

async def _cleanup_stale_managed_objects(self) -> None:
"""
Mark managed objects older than MANAGED_OBJECT_STALENESS_CUTOFF_DAYS days
in non-terminal states as 'stale_expired'. These will never complete and
should not be polled.
"""
cutoff = datetime.now(timezone.utc) - timedelta(days=MANAGED_OBJECT_STALENESS_CUTOFF_DAYS)
result = await self.prisma_client.db.litellm_managedobjecttable.update_many(
where={
"status": {"not_in": ["completed", "complete", "failed", "expired", "cancelled", "stale_expired"]},
"created_at": {"lt": cutoff},
},
data={"status": "stale_expired"},
)
if result > 0:
verbose_proxy_logger.warning(
f"CheckBatchCost: marked {result} stale managed objects "
f"(older than {MANAGED_OBJECT_STALENESS_CUTOFF_DAYS} days) as stale_expired"
)
Comment on lines +59 to +78

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Duplicate stale cleanup runs twice per poll cycle

CheckBatchCost._cleanup_stale_managed_objects() and CheckResponsesCost._cleanup_stale_managed_objects() are identical β€” both issue the exact same update_many against the full LiteLLM_ManagedObjectTable with no file_purpose filter. Since both jobs are scheduled at the same interval in proxy_server.py, every poll cycle triggers two separate full-table stale-row scans on the same set of rows.

The second run is a no-op (rows already marked stale_expired are excluded by the not_in filter), but it still incurs a full DB round-trip that re-scans the same large table you were trying to avoid loading into memory. At 336 K rows this doubles the remediation cost of the very problem being fixed.

Adding a file_purpose filter to each class's cleanup scopes the update to its own object type and eliminates the redundant scan:

# In CheckBatchCost._cleanup_stale_managed_objects
where={
    "file_purpose": "batch",
    "status": {"not_in": [...]},
    "created_at": {"lt": cutoff},
},

# In CheckResponsesCost._cleanup_stale_managed_objects
where={
    "file_purpose": "response",
    "status": {"not_in": [...]},
    "created_at": {"lt": cutoff},
},


async def check_batch_cost(self):
"""
Check if the batch JOB has been tracked.
Expand All @@ -70,14 +94,41 @@ async def check_batch_cost(self):
get_model_id_from_unified_batch_id,
)

await self._cleanup_stale_managed_objects()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unhandled exception in cleanup aborts the entire poll cycle

_cleanup_stale_managed_objects() is not wrapped in a try/except. If the cleanup update_many fails for any reason (DB timeout, transient connection error, schema issue), the exception propagates and the entire check_batch_cost() call is aborted β€” meaning pagination is never reached and no jobs are processed that cycle. The same issue exists in CheckResponsesCost.check_responses_cost() (line 63 of that file).

The cleanup is a best-effort operation and should not gate the main polling logic. Wrapping it in a try/except and logging the error (but continuing) would be safer:

try:
    await self._cleanup_stale_managed_objects()
except Exception as e:
    verbose_proxy_logger.warning(f"CheckBatchCost: stale cleanup failed, continuing: {e}")

The same fix should be applied in check_responses_cost.py around line 63.


# Look for all batches that have not yet been processed by CheckBatchCost
jobs = await self.prisma_client.db.litellm_managedobjecttable.find_many(
where={
"file_purpose": "batch",
"batch_processed" : False,
"status": {"not_in": ["failed", "expired", "cancelled"]}
}
)
try:
jobs = await self.prisma_client.db.litellm_managedobjecttable.find_many(
where={
"file_purpose": "batch",
"batch_processed": False,
"status": {"not_in": ["failed", "expired", "cancelled", "stale_expired"]},
},
take=MAX_OBJECTS_PER_POLL_CYCLE,
order={"created_at": "asc"},
)
except Exception:
# Fallback: batch_processed column may not exist on older schemas
verbose_proxy_logger.warning(
"CheckBatchCost: batch_processed column not found, querying without it"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bare except makes the fallback indistinguishable from a real DB failure

The warning message "batch_processed column not found, querying without it" is logged regardless of what exception was actually raised. If the primary query fails due to a connection timeout, an OOM condition from an oversized result, or another transient error, operators will see a benign-sounding schema-mismatch message instead of the real cause.

At minimum, log the original exception so the true error is visible:

Suggested change
except Exception:
# Fallback: batch_processed column may not exist on older schemas
verbose_proxy_logger.warning(
"CheckBatchCost: batch_processed column not found, querying without it"
except Exception as e:
# Fallback: batch_processed column may not exist on older schemas
verbose_proxy_logger.warning(
f"CheckBatchCost: primary query failed ({e!r}), retrying without batch_processed filter"
)

This makes genuine schema mismatches (PrismaClientKnownRequestError: column "batch_processed" does not exist) distinguishable from connection errors in logs, which is especially important given that this PR was motivated by hard-to-diagnose connection failures.

)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bare except Exception masks any DB error with a misleading schema message

The except Exception: block catches any exception from the find_many, not only a missing-column schema error. If the first query fails due to a DB connection issue, a timeout, or an OOM condition (the very problem this PR aims to fix), the log will say "batch_processed column not found" and the fallback query will be silently attempted. This can:

  1. Mislead operators diagnosing production incidents β€” the log implies a benign schema mismatch when the real cause is a connection failure.
  2. Mask a recurring DB error that silently swaps to the fallback every single cycle.

Consider catching a narrower exception (e.g., a Prisma PrismaClientKnownRequestError for the missing-column case) and re-raising for other error types. At minimum, log the original exception so the true error is visible:

except Exception as e:
    verbose_proxy_logger.warning(
        f"CheckBatchCost: primary query failed ({e}), retrying without batch_processed filter"
    )

jobs = await self.prisma_client.db.litellm_managedobjecttable.find_many(
where={
"file_purpose": "batch",
"status": {
"not_in": [
"failed",
"expired",
"cancelled",
"complete",
"completed",
"stale_expired",
]
},
},
take=MAX_OBJECTS_PER_POLL_CYCLE,
order={"created_at": "asc"},
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

stale_expired rows re-fetched in both the primary and fallback queries

_cleanup_stale_managed_objects() marks old rows as "stale_expired", but neither find_many query filters them out afterward:

  • Primary query (line 105): not_in: ["failed", "expired", "cancelled"] β€” "stale_expired" is absent.
  • Fallback query (line 118): not_in: ["failed", "expired", "cancelled", "complete", "completed"] β€” "stale_expired" is also absent.

This means rows just marked stale_expired are immediately re-fetched and processed in the same cycle, defeating the cleanup. CheckResponsesCost avoids this by using an allowlist ({"in": ["queued", "in_progress"]}), but CheckBatchCost uses a denylist that must be kept in sync. Both queries should add "stale_expired" to their not_in lists:

Suggested change
try:
jobs = await self.prisma_client.db.litellm_managedobjecttable.find_many(
where={
"file_purpose": "batch",
"batch_processed": False,
"status": {"not_in": ["failed", "expired", "cancelled"]},
},
take=MAX_OBJECTS_PER_POLL_CYCLE,
order={"created_at": "asc"},
)
except Exception:
# Fallback: batch_processed column may not exist on older schemas
verbose_proxy_logger.warning(
"CheckBatchCost: batch_processed column not found, querying without it"
)
jobs = await self.prisma_client.db.litellm_managedobjecttable.find_many(
where={
"file_purpose": "batch",
"status": {"not_in": ["failed", "expired", "cancelled", "complete", "completed"]},
},
take=MAX_OBJECTS_PER_POLL_CYCLE,
order={"created_at": "asc"},
)
jobs = await self.prisma_client.db.litellm_managedobjecttable.find_many(
where={
"file_purpose": "batch",
"batch_processed": False,
"status": {"not_in": ["failed", "expired", "cancelled", "stale_expired"]},
},
take=MAX_OBJECTS_PER_POLL_CYCLE,
order={"created_at": "asc"},
)
except Exception:
# Fallback: batch_processed column may not exist on older schemas
verbose_proxy_logger.warning(
"CheckBatchCost: batch_processed column not found, querying without it"
)
jobs = await self.prisma_client.db.litellm_managedobjecttable.find_many(
where={
"file_purpose": "batch",
"status": {"not_in": ["failed", "expired", "cancelled", "complete", "completed", "stale_expired"]},
},
take=MAX_OBJECTS_PER_POLL_CYCLE,
order={"created_at": "asc"},
)

for job in jobs:
# get the model from the job
unified_object_id = job.unified_object_id
Expand Down Expand Up @@ -163,14 +214,14 @@ async def check_batch_cost(self):

# Access content - handle both direct attribute and method call
if hasattr(_file_content, 'content'):
content_bytes = _file_content.content
content_bytes = _file_content.content # type: ignore[union-attr]
elif hasattr(_file_content, 'read'):
content_bytes = await _file_content.read()
content_bytes = await _file_content.read() # type: ignore[misc]
else:
content_bytes = _file_content
content_bytes = _file_content # type: ignore[assignment]

file_content_as_dict = _get_file_content_as_dictionary(
content_bytes
content_bytes # type: ignore[arg-type]
)

deployment_info = self.llm_router.get_deployment(model_id=model_id)
Expand All @@ -195,7 +246,7 @@ async def check_batch_cost(self):
file_content_dictionary=file_content_as_dict,
custom_llm_provider=llm_provider, # type: ignore
model_name=model_name,
model_info=deployment_model_info,
model_info=deployment_model_info, # type: ignore[arg-type]
)
)
logging_obj = LiteLLMLogging(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,15 @@
Cost tracking is handled automatically by litellm.aget_responses().
"""

from datetime import datetime, timedelta, timezone
from typing import TYPE_CHECKING

import litellm
from litellm._logging import verbose_proxy_logger
from litellm.constants import (
MANAGED_OBJECT_STALENESS_CUTOFF_DAYS,
MAX_OBJECTS_PER_POLL_CYCLE,
)

if TYPE_CHECKING:
from litellm.proxy.utils import PrismaClient, ProxyLogging
Expand All @@ -27,6 +32,26 @@ def __init__(
self.prisma_client: PrismaClient = prisma_client
self.llm_router: Router = llm_router

async def _cleanup_stale_managed_objects(self) -> None:
"""
Mark managed objects older than MANAGED_OBJECT_STALENESS_CUTOFF_DAYS days
in non-terminal states as 'stale_expired'. These will never complete and
should not be polled.
"""
cutoff = datetime.now(timezone.utc) - timedelta(days=MANAGED_OBJECT_STALENESS_CUTOFF_DAYS)
result = await self.prisma_client.db.litellm_managedobjecttable.update_many(
where={
"status": {"not_in": ["completed", "complete", "failed", "expired", "cancelled", "stale_expired"]},
"created_at": {"lt": cutoff},
},
data={"status": "stale_expired"},
)
if result > 0:
verbose_proxy_logger.warning(
f"CheckResponsesCost: marked {result} stale managed objects "
f"(older than {MANAGED_OBJECT_STALENESS_CUTOFF_DAYS} days) as stale_expired"
)

async def check_responses_cost(self):
"""
Check if background responses are complete and track their cost.
Expand All @@ -35,11 +60,15 @@ async def check_responses_cost(self):
- Cost is automatically tracked by litellm.aget_responses()
- Mark completed/failed/cancelled responses as complete in the database
"""
await self._cleanup_stale_managed_objects()

jobs = await self.prisma_client.db.litellm_managedobjecttable.find_many(
where={
"status": {"in": ["queued", "in_progress"]},
"file_purpose": "response",
}
},
take=MAX_OBJECTS_PER_POLL_CYCLE,
order={"created_at": "asc"},
)

verbose_proxy_logger.debug(f"Found {len(jobs)} response jobs to check")
Expand Down
9 changes: 9 additions & 0 deletions litellm/constants.py
Original file line number Diff line number Diff line change
Expand Up @@ -1351,6 +1351,15 @@
os.getenv("PROXY_BUDGET_RESCHEDULER_MIN_TIME", 597)
)
PROXY_BATCH_POLLING_INTERVAL = int(os.getenv("PROXY_BATCH_POLLING_INTERVAL", 3600))
MAX_OBJECTS_PER_POLL_CYCLE = int(os.getenv("MAX_OBJECTS_PER_POLL_CYCLE", 50))
MANAGED_OBJECT_STALENESS_CUTOFF_DAYS = int(
os.getenv("MANAGED_OBJECT_STALENESS_CUTOFF_DAYS", 7)
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Zero or negative env values silently destroy all active jobs

MANAGED_OBJECT_STALENESS_CUTOFF_DAYS and MAX_OBJECTS_PER_POLL_CYCLE are parsed with a bare int() but have no lower-bound validation.

  • If MANAGED_OBJECT_STALENESS_CUTOFF_DAYS=0 is set, cutoff = datetime.now(timezone.utc) - timedelta(days=0) equals now. Every row satisfies created_at < now, so all non-terminal managed objects are immediately bulk-updated to stale_expired on the first poll cycle β€” silently discarding all in-flight batches and responses.
  • If MAX_OBJECTS_PER_POLL_CYCLE=0 is set, take=0 is passed to Prisma's find_many. Prisma's behaviour with take=0 is undefined/implementation-dependent (some versions return zero rows, others may error), so the polling loop silently stops processing jobs.

Both values should be clamped to a sensible minimum:

Suggested change
MAX_OBJECTS_PER_POLL_CYCLE = int(os.getenv("MAX_OBJECTS_PER_POLL_CYCLE", 50))
MANAGED_OBJECT_STALENESS_CUTOFF_DAYS = int(
os.getenv("MANAGED_OBJECT_STALENESS_CUTOFF_DAYS", 7)
)
MAX_OBJECTS_PER_POLL_CYCLE = max(1, int(os.getenv("MAX_OBJECTS_PER_POLL_CYCLE", 50)))
MANAGED_OBJECT_STALENESS_CUTOFF_DAYS = max(1, int(
os.getenv("MANAGED_OBJECT_STALENESS_CUTOFF_DAYS", 7)
))

# Set PROXY_BATCH_POLLING_ENABLED=false to disable the CheckBatchCost and
# CheckResponsesCost background polling jobs entirely (e.g. to avoid DB load on
# installations with large numbers of stale managed objects).
_batch_polling_env = os.getenv("PROXY_BATCH_POLLING_ENABLED", "true").lower()
PROXY_BATCH_POLLING_ENABLED = _batch_polling_env == "true"
PROXY_BUDGET_RESCHEDULER_MAX_TIME = int(
os.getenv("PROXY_BUDGET_RESCHEDULER_MAX_TIME", 605)
)
Expand Down
7 changes: 4 additions & 3 deletions litellm/proxy/proxy_server.py
Original file line number Diff line number Diff line change
Expand Up @@ -214,6 +214,7 @@ def generate_feedback_box():
DEFAULT_MODEL_CREATED_AT_TIME,
LITELLM_PROXY_ADMIN_NAME,
PROMETHEUS_FALLBACK_STATS_SEND_TIME_HOURS,
PROXY_BATCH_POLLING_ENABLED,
PROXY_BATCH_POLLING_INTERVAL,
PROXY_BATCH_WRITE_AT,
PROXY_BUDGET_RESCHEDULER_MAX_TIME,
Expand Down Expand Up @@ -260,7 +261,6 @@ def generate_feedback_box():
claude_code_marketplace_router,
)
from litellm.proxy.anthropic_endpoints.endpoints import router as anthropic_router
from litellm.proxy.realtime_endpoints.endpoints import router as webrtc_router
from litellm.proxy.anthropic_endpoints.skills_endpoints import (
router as anthropic_skills_router,
)
Expand Down Expand Up @@ -471,6 +471,7 @@ def generate_feedback_box():
from litellm.proxy.prompts.prompt_endpoints import router as prompts_router
from litellm.proxy.public_endpoints import router as public_endpoints_router
from litellm.proxy.rag_endpoints.endpoints import router as rag_router
from litellm.proxy.realtime_endpoints.endpoints import router as webrtc_router
from litellm.proxy.rerank_endpoints.endpoints import router as rerank_router
from litellm.proxy.response_api_endpoints.endpoints import router as response_router
from litellm.proxy.route_llm_request import route_request
Expand Down Expand Up @@ -6069,7 +6070,7 @@ async def initialize_scheduled_background_jobs( # noqa: PLR0915
"Invalid maximum_spend_logs_retention_interval value"
)
### CHECK BATCH COST ###
if llm_router is not None:
if llm_router is not None and PROXY_BATCH_POLLING_ENABLED:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PROXY_BATCH_POLLING_ENABLED controls responses polling too β€” name is misleading

The kill-switch disables both CheckBatchCost and CheckResponsesCost jobs, but the env var is named PROXY_BATCH_POLLING_ENABLED. A user who wants to disable only responses polling (to investigate a response-specific OOM) would have no targeted knob β€” setting this flag kills batch polling as well.

Similarly, a user reading PROXY_BATCH_POLLING_ENABLED=false in their config 6 months later might assume responses polling is unaffected.

Consider either:

  • Renaming to PROXY_MANAGED_OBJECT_POLLING_ENABLED to accurately reflect both job types, or
  • Adding a separate PROXY_RESPONSES_POLLING_ENABLED flag so each job can be controlled independently

The documentation entry in config_settings.md should be updated to match whichever approach is chosen.

try:
from litellm_enterprise.proxy.common_utils.check_batch_cost import (
CheckBatchCost,
Expand Down Expand Up @@ -6100,7 +6101,7 @@ async def initialize_scheduled_background_jobs( # noqa: PLR0915
pass

### CHECK RESPONSES COST ###
if llm_router is not None:
if llm_router is not None and PROXY_BATCH_POLLING_ENABLED:
try:
from litellm_enterprise.proxy.common_utils.check_responses_cost import (
CheckResponsesCost,
Expand Down
Loading
Loading