Skip to content

fix(ci): tell stale branches to pull latest staging on vacuous type-check runs - #36117

Open
mateo-berri wants to merge 3 commits into
litellm_internal_stagingfrom
litellm_vacuous_gate_staging_hint
Open

fix(ci): tell stale branches to pull latest staging on vacuous type-check runs#36117
mateo-berri wants to merge 3 commits into
litellm_internal_stagingfrom
litellm_vacuous_gate_staging_hint

Conversation

@mateo-berri

Copy link
Copy Markdown
Contributor

TLDR

Problem this solves:

  • Vacuous type-check gate failures confuse OSS contributors on stale bases
  • The fix is already on staging, but the message never says so
  • Contributors misdiagnose CI as broken and block on maintainers

How it solves it:

  • The FAIL message now says to pull latest litellm_internal_staging
  • Regression test pins the exact remedy text

User Flow

Before, the failing check tells the contributor a crash happened but not what to do about it, so they stall and ask maintainers for help

  1. A contributor branches off a weeks-old litellm_internal_staging, fixes a bug, and opens a PR at https://github.com/BerriAI/litellm/compare, then waits for CI
  2. They open the PR's checks tab, GET https://github.com/BerriAI/litellm/pull/{number}/checks, and see the lint workflow red
  3. They click into the failed step's log and read: "FAIL: basedpyright produced no errors, but basedpyright-code-budget.json allows up to ~206305. The type checker almost certainly crashed or emitted nothing; refusing to certify a vacuous run."
  4. Nothing in the message says what to change, so they re-run basedpyright locally (where it passes on their machine), conclude the CI pipeline itself is broken, and post a comment asking maintainers to re-run the job and fix the step
  5. The PR sits blocked until a maintainer replies in Slack or on the PR telling them to update their branch

After, the same failure names the remedy in the log itself, so the contributor unblocks themselves without waiting on a maintainer

  1. A contributor branches off a weeks-old litellm_internal_staging, fixes a bug, and opens a PR at https://github.com/BerriAI/litellm/compare, then waits for CI
  2. They open the PR's checks tab, GET https://github.com/BerriAI/litellm/pull/{number}/checks, and see the lint workflow red
  3. They click into the failed step's log and read the same FAIL line, now ending with: "This usually means this branch is missing type checker fixes from its base branch: pull the latest litellm_internal_staging and merge or rebase onto it."
  4. They run git fetch origin litellm_internal_staging, merge or rebase onto it, and push
  5. CI re-runs on the updated branch and the type-check gate goes green with no maintainer involvement

Relevant issues

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have added meaningful tests
  • My PR passes all CI/CD checks (e.g., lint, format, unit tests)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

The gate fires on the "basedpyright exits cleanly while emitting nothing" condition, which in the wild comes from an OOM crash on a stale base. To reproduce that condition deterministically without a multi-gigabyte stale checkout, both legs run the real gate CLI against the repo with the gate-owned venv's basedpyright entry point replaced by a stub that prints an empty diagnostics payload and exits 0, which is byte-for-byte the input the gate sees in the real incident

Before, litellm_internal_staging at f6587fa: the failure states the problem and stops, with no remedy

$ printf '#!/bin/sh\necho "{\"generalDiagnostics\": []}"\n' > .venv-typecheck/bin/basedpyright
$ uv run --no-sync python scripts/type_check_gate.py --base origin/litellm_internal_staging; echo "exit=$?"
FAIL: basedpyright produced no errors, but basedpyright-code-budget.json allows up to ~206305. The type checker almost certainly crashed or emitted nothing; refusing to certify a vacuous run.
exit=1

After, this PR at 5741b26: the same run now ends with the remedy a stale-base contributor needs

$ uv run --no-sync python scripts/type_check_gate.py --base origin/litellm_internal_staging; echo "exit=$?"
FAIL: basedpyright produced no errors, but basedpyright-code-budget.json allows up to ~206305. The type checker almost certainly crashed or emitted nothing; refusing to certify a vacuous run. This usually means this branch is missing type checker fixes from its base branch: pull the latest litellm_internal_staging and merge or rebase onto it.
exit=1

This mirrors the exact confusion an OSS contributor reported on #35952, where they read the old message, concluded the CI step itself was broken, and asked maintainers to re-run the job instead of updating their base

Type

🚄 Infrastructure

Changes

scripts/type_check_gate.py appends one sentence to the vacuous-run FAIL message in cmd_check, deriving the branch name from DEFAULT_BASE so the hint can never drift from the gate's actual base. tests/test_litellm/test_type_check_gate.py adds a regression test that drives cmd_check with empty counts and asserts both the vacuous-run refusal and the exact user-visible remedy string, so dropping or rewording the hint fails the suite

Final Attestation

  • The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

@greptile-apps

greptile-apps Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR improves the vacuous type-check failure message so contributors are directed to update from litellm_internal_staging.

  • Appends an actionable merge-or-rebase remedy to the existing failure diagnostic.
  • Adds a regression test that pins both the refusal text and the new remedy.

Confidence Score: 5/5

The PR appears safe to merge.

The change only augments an existing CI failure diagnostic and adds focused coverage for the intended remedy, with no blocking or independently actionable issue remaining.

Important Files Changed

Filename Overview
scripts/type_check_gate.py Extends the vacuous-run diagnostic with the intended staging update instructions; no actionable defect was established.
tests/test_litellm/test_type_check_gate.py Adds a focused regression test covering the failure condition and exact user-facing remedy.

Reviews (1): Last reviewed commit: "fix(ci): tell stale branches to pull lat..." | Re-trigger Greptile

@ryan-crabbe-berri ryan-crabbe-berri left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you drop the test

@codecov

codecov Bot commented Aug 6, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants