test(e2e): skip the strict-priority and throughput SLO tests pending LIT-5118 / LIT-5119

[ryan-crabbe-berri]

TLDR

Problem this solves:

• two tests fail every stage run for environment reasons, not code bugs
• strict-priority e2e: stage proxy lacks the dynamic_rate_limiter_v3 config it requires
• throughput SLO test: 65.9% of requests die at the ELB as 502/503

How it solves it:

• skip both with reasons naming their tickets, LIT-5118 and LIT-5119
• skips hand the registry cells back to the gap list, so nothing reads as covered

Relevant issues

Linear ticket

Refs LIT-5118, LIT-5119

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Both failures are from the stage run 2026-08-02 00:52 UTC (pod litellm-e2e-1-0-0-main-20260802005210-s4d6g, image at ba480a6), the run's only failures besides the budget reset flake fixed in #35572

E  AssertionError: dev key was still served at 286 tokens, past the saturation threshold (200)
   and 100-token dev reservation; strict priority enforcement never engaged.
   quota_management/ratelimit/test_dynamic_rate_limit_priority_e2e.py:218

E  AssertionError: 6354/9646 requests failed (65.9% > 1.0% allowed); 6130x /chat/completions:
   LocustBadStatusCode(code=503); 224x /chat/completions: LocustBadStatusCode(code=502)
   load/test_chat_completions_throughput_e2e.py:66

The strict-priority failure is config, not code: Loki across every gateway and backend pod in the window has zero lines matching dynamic_rate_limiter|priority_reservation|Priority-based (73,922 scanned), so the limiter callback is not running on stage at all. The load failure is the LIT-5054 capacity story on its other assertion; the requests died at the ELB before reaching a pod

After (at the PR commit), both deselect cleanly with no proxy involved:

quota_management/.../test_strict_mode_blocks_saturated_priority_but_serves_the_other SKIPPED [ 50%]
load/test_chat_completions_throughput_e2e.py::...::test_sustains_throughput_slo_under_load SKIPPED [100%]
============================== 2 skipped in 0.02s ==============================

Type

✅ Test

Changes

Adds a @pytest.mark.skip naming the blocking ticket to test_strict_mode_blocks_saturated_priority_but_serves_the_other (LIT-5118: stage needs the dynamic_rate_limiter_v3 callbacks plus priority_reservation settings in the infra repo config) and to test_sustains_throughput_slo_under_load (LIT-5119: stage refuses the closed-loop load at the ELB until the fleet is pre-scaled for the load phase). No assertion changes. Per the coverage-registry rules a skipped test returns its cell to the gap list, so quota_management.ratelimit.priority_strict.picks_under_tpm and reliability.perf.throughput.under_slo will report as uncovered rather than green

The generous-mode sibling stays enabled: it asserts the absence of a 429 below the saturation threshold, which holds with or without the limiter, so it cannot go red for this config gap (noted in LIT-5118 that its green is not evidence the feature works on stage)

QA runbook

• tests/e2e/quota_management/ratelimit/test_dynamic_rate_limit_priority_e2e.py::TestDynamicRateLimitPriority::test_strict_mode_blocks_saturated_priority_but_serves_the_other - now skipped pending LIT-5118
  • Run pytest on the file and expect the strict test to report SKIPPED with the LIT-5118 reason while the generous test still runs
  • Sanity check: this test makes sense and is not hand-wavey or potentially flaky
• tests/e2e/load/test_chat_completions_throughput_e2e.py::TestChatCompletionsThroughput::test_sustains_throughput_slo_under_load - now skipped pending LIT-5119
  • Run pytest on the file and expect the SLO test to report SKIPPED with the LIT-5119 reason
  • Sanity check: this test makes sense and is not hand-wavey or potentially flaky

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[greptile-apps]

Greptile Summary

This PR temporarily skips two environment-blocked end-to-end tests while preserving honest coverage-registry reporting.

• Skips the chat-completions throughput SLO test pending stage fleet pre-scaling under LIT-5119.
• Skips the strict-priority rate-limit test pending required stage limiter configuration under LIT-5118.
• Leaves assertions and the independently scoped generous-priority test unchanged.

Confidence Score: 5/5

The PR appears safe to merge because it makes only the explicitly intended test skips and does not falsely preserve their coverage status.

The coverage collector recognizes both skipped tests as uncovered, decorator order does not alter that behavior, and the remaining generous-priority test covers a separate scenario as described.

Important Files Changed

Filename	Overview
tests/e2e/load/test_chat_completions_throughput_e2e.py	Adds a ticket-referenced unconditional skip to the environment-blocked throughput SLO test; skipped coverage is correctly returned to the registry gap list.
tests/e2e/quota_management/ratelimit/test_dynamic_rate_limit_priority_e2e.py	Adds a ticket-referenced unconditional skip to the strict-priority test while retaining the distinct generous-mode test and honest registry accounting.

_{Reviews (1): Last reviewed commit: "test(e2e): skip the strict-priority and ..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!