fix(alerting): dedupe scheduled Slack spend reports across pods

[ryan-crabbe-berri]

TLDR

Problem this solves:

• Every pod ran its own scheduled Slack spend report jobs
• Multi-replica and multi-worker deployments got one duplicate report per pod
• Every deploy re-fired the weekly report on every pod

How it solves it:

• Scheduled sends now race for the shared PodLockManager redis lock
• The lock is never released; its TTL marks the window as sent
• Gates are non-reentrant: a live lock blocks its own holder too
• Weekly TTL spans the reporting window minus an hour of margin
• Covers weekly, monthly, prometheus fallback stats, and daily report loop

User Flow

Before: an ML platform team running the proxy with several replicas gets every Slack spend report several times

1. The admin sets alerting: ["slack"], spend_report_frequency: "1d" and deploys 3 replicas
2. Their Slack channel receives the same spend report three times, minutes apart
3. Each redeploy triggers another burst of duplicates

After: the same config produces exactly one report per window

1. The admin wires redis (litellm_settings.cache, coordination_redis, or use_redis_transaction_buffer) and deploys the same 3 replicas
2. The Slack channel receives the spend report exactly once; the other replicas log that another pod holds the lock
3. Redeploys mid-window send nothing extra

Relevant issues

Fixes #14809

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

Two proxy instances simulating two pods (before run on ports 4113/4114, after run on 4115/4116), sharing one Postgres and one redis container that stayed up for every run; the runs differ only in whether litellm is wired to that redis. Config: alerting: ["slack"], spend_report_frequency: "1d", plus a litellm_settings.cache redis block in the wired variant only. Spend was seeded with a real groq chat completion through instance A; the weekly job fires 10-310s after each instance boots

python webhook_catcher.py 9113 | tee catcher.log &
for port in 4113 4114; do   # after run: 4115 4116 with qa14809_config.yaml (adds the redis cache block)
  SLACK_WEBHOOK_URL=http://127.0.0.1:9113/slack LITELLM_LOG=INFO \
    python -m litellm.proxy.proxy_cli --config qa14809_config_noredis.yaml --port $port &
done
curl -sS http://127.0.0.1:4113/v1/chat/completions -H "Authorization: Bearer $MASTER_KEY" \
  -d '{"model": "groq-llama-3.3-70b", "messages": [{"role": "user", "content": "hi"}]}'

Before, at 363d56f, redis running but not wired into litellm: each pod sends its own copy of the same report

2026-08-11T00:14:29Z POST /slack text=*Spend Report for `08-09-2026 - 08-10-2026` (1 days)* ...
2026-08-11T00:18:09Z POST /slack text=*Spend Report for `08-09-2026 - 08-10-2026` (1 days)* ...

After, at 3f0a793, redis wired via litellm_settings.cache: exactly one report, and a 2s loop deleted the pre-existing weekly_spend_report_sent_* cache key throughout the run, so the old date-window dedupe could not be what suppressed the second send

catcher:    2026-08-11T00:37:57Z POST /slack text=*Spend Report for `08-09-2026 - 08-10-2026` (1 days)* ...
pod A log:  17:37:53 INFO pod_lock_manager.py:78  - Pod 1c32c032-... successfully acquired Redis lock for cronjob_id=weekly_spend_report_job
pod B log:  17:40:46 INFO pod_lock_manager.py:100 - Pod 8d81e64b-... could not acquire lock for cronjob_id=weekly_spend_report_job, held by pod 1c32c032-...

The review-found TTL boundary fix is confirmed by reading the lock back right after acquisition: redis-cli ttl cronjob_lock:weekly_spend_report_job returned 82794 (the 1d window minus the hour margin, minus 7s elapsed); the same read against the pre-fix TTL gave 86391, i.e. a TTL equal to the job interval, which is the boundary defect described above

Same fixed commit with redis unwired still duplicates (both pods proceed on the no-redis path), which is the first caveat below

2026-08-11T00:42:14Z POST /slack text=*Spend Report for `08-09-2026 - 08-10-2026` (1 days)* ...
2026-08-11T00:45:49Z POST /slack text=*Spend Report for `08-09-2026 - 08-10-2026` (1 days)* ...

Commits after 3f0a793 address Greptile findings: the startup prometheus fallback send now routes through the same gated closure, and gates pass allow_reentrant=False so a pod booting within an hour of the fallback cron cannot send twice; both are pinned by unit tests asserting the exact lock calls

Type

🐛 Bug Fix

Caveats (if any)

• Cross-pod dedupe requires redis wired via cache, coordination_redis, or use_redis_transaction_buffer
• A redis outage at fire time skips that window's report (fail-closed, like all locked cron jobs)
• Mid-window deploys no longer re-send the report; the boot-time re-fire was the bug
• spend_report_frequency: "0d" now fails startup; it used to coerce to an every-second schedule

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

---

Note

^{Cursor Bugbot is generating a summary for commit 1884cc4. Configure here.}

[greptile-apps]

Greptile Summary

The PR coordinates scheduled Slack spend reports across replicas using shared Redis locks

• Adds stable lock identifiers for weekly, monthly, Prometheus fallback, and daily reports
• Adds non-reentrant lock acquisition so a live window lock blocks its own holder as well as other pods
• Routes both Prometheus startup and cron sends through the same lock-gated wrapper
• Adds regression coverage for same-pod reacquisition and scheduled report deduplication

Confidence Score: 5/5

The PR appears safe to merge

No blocking failure remains

Important Files Changed

Filename	Overview
litellm/proxy/proxy_server.py	Scheduled report wrappers consistently acquire non-reentrant shared locks, including both Prometheus startup and cron paths
litellm/proxy/db/db_transaction_queue/pod_lock_manager.py	Adds explicit non-reentrant semantics while preserving existing leader-election behavior by default
litellm/integrations/SlackAlerting/slack_alerting.py	Routes eligible daily reports through the shared window lock before sending
tests/test_litellm/proxy/proxy_server/test_lifecycle.py	Covers lock-held, lock-free, and unavailable-lock behavior for scheduled report jobs
tests/test_litellm/proxy/db/db_transaction_queue/test_pod_lock_manager.py	Verifies that non-reentrant acquisition rejects a live lock held by the same pod

_{Reviews (3): Last reviewed commit: "fix(alerting): make report lock acquisit..." | Re-trigger Greptile}

[codecov]

Codecov Report

❌ Patch coverage is 78.12500% with 7 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...tellm/integrations/SlackAlerting/slack_alerting.py	22.22%	7 Missing ⚠️

📢 Thoughts on this report? Let us know!

[ryan-crabbe-berri]

@greptileai re-review please: the startup prometheus fallback send now goes through the same pod lock, tests updated accordingly

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_fix_duplicate_spend_reports (1884cc4) with litellm_internal_staging (02b0ee7)}

[ryan-crabbe-berri]

@greptileai re-review: report gates now pass allow_reentrant=False so a live lock also blocks its own holder within the window

[ryan-crabbe-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 1884cc4. Configure here.}