Skip to content

test(e2e): drop xfail markers for now-fixed team-budget-JSON and custom-pricing-leak bugs - #31249

Merged
mubashir1osmani merged 1 commit into
litellm_internal_stagingfrom
litellm_drop_resolved_xfail_markers
Jun 25, 2026
Merged

test(e2e): drop xfail markers for now-fixed team-budget-JSON and custom-pricing-leak bugs#31249
mubashir1osmani merged 1 commit into
litellm_internal_stagingfrom
litellm_drop_resolved_xfail_markers

Conversation

@mubashir1osmani

Copy link
Copy Markdown
Collaborator

Relevant issues

Two e2e tests were xfail(strict=True) for known proxy bugs that are now fixed on litellm_internal_staging:

Pre-Submission checklist

  • I have added meaningful tests
  • My PR's scope is as isolated as possible; it only solves 1 specific problem

Screenshots / Proof of Fix

Both fixes already shipped to litellm_internal_staging, so the two tests pass their assertions there. Because they were still marked xfail(strict=True), the run reported the unexpected passes as failures:

[XPASS(strict)] known proxy bug: /team/new writes budget_limits straight to the Json? column ...
[XPASS(strict)] known proxy bug: a deployment's custom per-token pricing leaks into the shared cost map ...
FAILED budgets/test_team_multi_window_budget_e2e.py::test_team_short_window_blocks_then_resets
FAILED llm_translation/test_custom_pricing_e2e.py::test_custom_pricing_is_isolated_from_sibling_deployment
=================== 2 failed, 59 passed in 497.24s ===================

With the strict markers removed, both run as ordinary passing tests, taking the suite to all-green; their assertions are unchanged, so they now stand as regression guards for the two fixes

Type

🧹 Refactoring
✅ Test

Changes

Removed the @pytest.mark.xfail(strict=True, ...) decorator from test_team_short_window_blocks_then_resets and test_custom_pricing_is_isolated_from_sibling_deployment, as each marker's reason instructed once the underlying bug was fixed. Updated both docstrings to describe the regression each test now pins rather than the bug it used to tolerate. No assertion changed

…custom-pricing-leak bugs

Both tests were xfail(strict=True) for known proxy bugs: /team/new writing
budget_limits as a raw list (Prisma 500) and custom per-token pricing leaking into
the shared cost map for sibling deployments. Both are fixed, so the tests pass and
strict mode reports the unexpected pass as a failure. Remove the markers (as their
reasons instructed) so they run as plain regression guards; docstrings updated to
describe the regression each now pins.
@greptile-apps

greptile-apps Bot commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR promotes two previously-skipped e2e tests from xfail(strict=True) to ordinary passing tests now that the underlying proxy bugs have been fixed on litellm_internal_staging. No assertions or test logic were changed — only the decorator and docstrings.

  • Removes the xfail marker from test_team_short_window_blocks_then_resets, which now acts as a regression guard ensuring /team/new correctly json.dumps the budget window list before writing to the Json? column.
  • Removes the xfail marker from test_custom_pricing_is_isolated_from_sibling_deployment, which now guards against custom per-token pricing leaking into the shared cost map for sibling deployments of the same underlying model.

Confidence Score: 5/5

Safe to merge — only xfail markers and docstrings are changed, no test logic or assertions are touched.

Both tests retain all their original assertions and now run unconditionally as regression guards. The PR description references the concrete XPASS output that triggered this change, satisfying the team's evidence-of-fix rule. No production code is modified.

No files require special attention.

Important Files Changed

Filename Overview
tests/e2e/budgets/test_team_multi_window_budget_e2e.py Removes xfail(strict=True) marker from test_team_short_window_blocks_then_resets and updates the module docstring to describe it as a regression guard for the now-fixed /team/new json.dumps bug; no assertions changed.
tests/e2e/llm_translation/test_custom_pricing_e2e.py Removes xfail(strict=True) marker from test_custom_pricing_is_isolated_from_sibling_deployment and updates the module docstring to describe it as a regression guard for the now-fixed pricing isolation bug; no assertions changed.

Reviews (1): Last reviewed commit: "test(e2e): drop xfail markers for the no..." | Re-trigger Greptile

@mubashir1osmani
mubashir1osmani enabled auto-merge (squash) June 24, 2026 22:44
@mubashir1osmani

Copy link
Copy Markdown
Collaborator Author

these two tests were internally found bugs that are now fixed on litellm_internal_staging - removing xfail markers

@codecov

codecov Bot commented Jun 24, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@mubashir1osmani
mubashir1osmani merged commit 1ff1557 into litellm_internal_staging Jun 25, 2026
122 checks passed
@mubashir1osmani
mubashir1osmani deleted the litellm_drop_resolved_xfail_markers branch June 25, 2026 01:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants