Skip to content
Merged
Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
9f927b6
ci(lint): enforce type-discipline budget for casts and type guards
mateo-berri Jun 16, 2026
bcab7f5
ci(lint): enforce suppression-reason budgets and guard budgets agains…
mateo-berri Jun 16, 2026
626a302
ci(lint): ban mutable collections in annotations and all mutable cons…
mateo-berri Jun 16, 2026
7f64fa1
ci(lint): recommend pydantic at boundaries and add functional-refacto…
mateo-berri Jun 16, 2026
0420802
style: make CLAUDE.md more concise
mateo-berri Jun 16, 2026
957c30d
chore: update CLAUDE.md guidelines
mateo-berri Jun 16, 2026
3851419
ci(lint): renumber mutable construction LIT009 -> LIT002 next to LIT001
mateo-berri Jun 16, 2026
23e2aff
fix: numbering in CLAUDE.md
mateo-berri Jun 16, 2026
5c18687
test(lint): test type-discipline checker, scope LIT007 to return types
mateo-berri Jun 16, 2026
19c8afa
fix(lint): best-effort worktree teardown so cleanup can't mask the re…
mateo-berri Jun 16, 2026
5f5f8bb
fix(lint): ratchet fails loudly on an unresolvable base; drop dead ch…
mateo-berri Jun 16, 2026
7d58e15
fix(lint): degrade malformed source to LIT000 instead of crashing the…
mateo-berri Jun 16, 2026
827b4ba
perf(lint): skip the base worktree scan when no rule is over its ceiling
mateo-berri Jun 16, 2026
0ac778b
fix(lint): exempt .dict()/.list()/.set() method calls from LIT002
mateo-berri Jun 16, 2026
45f64ab
Merge litellm_internal_staging into litellm_type_discipline_gate
mateo-berri Jun 16, 2026
127de03
chore(lint): bump basedpyright ceilings to absorb staging base drift
mateo-berri Jun 16, 2026
f3ca65c
fix(lint): ratchet guard watches every budget file, not just two
mateo-berri Jun 16, 2026
ded1c54
Merge branch 'litellm_internal_staging' into litellm_type_discipline_…
mateo-berri Jun 16, 2026
a19f4e3
fix: add a lot more slack
mateo-berri Jun 16, 2026
ad6f016
fix(lint): restore LIT003 frozen slack to 0
mateo-berri Jun 16, 2026
83d9287
fix(lint): restore documented slack 10 for the buffered LIT rules
mateo-berri Jun 16, 2026
e0c33a3
fix(lint): ratchet LIT003 baseline down to its actual count
mateo-berri Jun 16, 2026
8efee05
fix: increase slack
mateo-berri Jun 16, 2026
ad80477
fix: increase slack
mateo-berri Jun 16, 2026
73620e6
docs(lint): align gate docstring with buffered LIT003/LIT004 slack
mateo-berri Jun 16, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions .github/workflows/test-linting.yml
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,12 @@ jobs:
run: |
uv run --no-sync python scripts/ruff_strict_gate.py --base "$BASE_SHA"

- name: Check type-discipline budget (mutable collections / casts / type guards / kwargs / unexplained suppressions, delta vs base)
env:
BASE_SHA: ${{ github.event.pull_request.base.sha }}
run: |
uv run --no-sync python scripts/type_discipline_gate.py --base "$BASE_SHA"

- name: Print OpenAI version
run: |
uv run --no-sync python -c "import openai; print(f'OpenAI version: {openai.__version__}')"
Expand All @@ -93,6 +99,33 @@ jobs:
run: |
uv run --no-sync python -c "from litellm import *" || (echo '🚨 import failed, this means you introduced unprotected imports! 🚨'; exit 1)

# Intentionally NON-GATING. This job turns red when a *-budget.json ceiling is
# raised (or a rule/budget is dropped) so a loosening is obvious in review, but it
# must be kept OUT of the branch-protection required-checks list so a justified
# bump can still be merged by a human who has seen and accepted the red.
budget-ratchet:
runs-on: ubuntu-latest
timeout-minutes: 5
permissions:
contents: read

steps:
- uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
with:
fetch-depth: 0
persist-credentials: false

- name: Set up Python
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
with:
python-version: "3.12"

- name: Ratchet check (budgets may only decrease; non-gating)
env:
BASE_SHA: ${{ github.event.pull_request.base.sha }}
run: |
python scripts/budget_ratchet_check.py --base "$BASE_SHA"

secret-scan:
runs-on: ubuntu-latest
timeout-minutes: 5
Expand Down
2 changes: 2 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,8 @@ Run tests, format your code, and lint your code before each commit

When you fix strict-rule violations gated by `ruff-strict-budget.json`, run `make lint-strict-budget-update` and commit the lowered baselines so the ceilings ratchet down instead of leaving stale headroom

If you get an LIT001 or LIT002 fail, refactor the code to follow functional programming best practices rather than introducing mutable data structures. For example, build values in one shot with comprehensions or generators wrapped in `tuple()` / `frozenset()` instead of seeding an empty `list`/`dict`/`set` and mutating it over time. Ideally `# mutable-ok` is never used; reach for it only as a genuine last resort when an immutable rewrite is truly impossible, and always pair it with a real reason

Ask to commit and push your work when you're done (or if you're confident that your code is good and works, just do it)

When you must use real LLM models to, for example, write e2e tests, write a QA runbook, etc., make sure to use the latest models (doesn't have to be smartest, can also be a modern small, fast one. No strong preference for smart vs fast here, just use something modern) as of the year and month of the current date. Do a web search as necessary to figure that out
Expand Down
2 changes: 1 addition & 1 deletion ruff-strict-budget.json
Original file line number Diff line number Diff line change
Expand Up @@ -8,5 +8,5 @@
"PLR0913": { "baseline": 1813, "slack": 3 },
"PLW0603": { "baseline": 183, "slack": 3 },
"RUF012": { "baseline": 158, "slack": 3 },
"TID251": { "baseline": 2404, "slack": 10 }
"TID251": { "baseline": 2662, "slack": 10 }
}
14 changes: 13 additions & 1 deletion ruff-strict.toml
Original file line number Diff line number Diff line change
Expand Up @@ -17,4 +17,16 @@ max-args = 5
"typing.Dict".msg = "Frozen dataclass / NamedTuple / ReadOnly TypedDict; create a Mapping alias with concrete value types if truly dynamic."
"typing.Set".msg = "frozenset[X] or AbstractSet[X]."
"typing.MutableSequence".msg = "Sequence[X]."
"typing.MutableMapping".msg = "See typing.Dict."
"typing.MutableMapping".msg = "See typing.Dict."
# Unchecked casts: cast() lies to the type checker with no runtime guarantee.
# Validate into a concrete frozen type at the boundary (pydantic) instead.
# Per-call-site coverage lives in check_type_discipline.py (LIT006); this freezes
# new cast imports. Suppress (with a reason) via `# noqa: TID251 # <reason>`.
"typing.cast".msg = "No unchecked casts: validate into a frozen dataclass/NamedTuple/ReadOnly TypedDict at the boundary (pydantic)."
"typing_extensions.cast".msg = "Same as typing.cast."
# Unverified narrowing predicates: the checker never validates the guard body, so a
# wrong guard silently corrupts types. Banned outright (there are none today).
"typing.TypeGuard".msg = "Unverified narrowing. Parse into a concrete type, or use isinstance for a runtime-checked narrowing."
"typing_extensions.TypeGuard".msg = "Same as typing.TypeGuard."
"typing.TypeIs".msg = "Unverified narrowing (the body is trusted). Parse into a concrete type instead."
"typing_extensions.TypeIs".msg = "Same as typing.TypeIs."
139 changes: 139 additions & 0 deletions scripts/budget_ratchet_check.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,139 @@
#!/usr/bin/env python3
"""Non-gating ratchet guard: budget ceilings may only fall, never rise.

The `*-budget.json` files (ruff-strict-budget.json, type-discipline-budget.json) are
one-way ratchets: each rule's ceiling is `baseline + slack`, and the whole point is
to drive that number DOWN over time. This check compares every budget file against
its own content at the merge-base with the target branch and fails (exits 1, red) if:

* a rule's ceiling went up,
* a rule was dropped from a budget (its ceiling effectively became infinite), or
* an entire budget file was deleted.

New rules and lowered/equal ceilings are fine.

This is deliberately NOT a gating check. It should turn the run red so that a
loosening is impossible to miss in review, but it must stay OUT of the
branch-protection required-checks list: a justified bump (e.g. banning a new API,
which mechanically raises a baseline) can then still be merged by a human who has
seen the red and accepted it.

Usage:
python scripts/budget_ratchet_check.py [--base REF] [budget.json ...]

Stdlib only.
"""

from __future__ import annotations

import argparse
import json
import subprocess
import sys
from pathlib import Path
from typing import NamedTuple

REPO_ROOT = Path(__file__).resolve().parent.parent
DEFAULT_BASE = "origin/litellm_internal_staging"
DEFAULT_BUDGETS: tuple[str, ...] = (
"ruff-strict-budget.json",
"type-discipline-budget.json",
)


class Regression(NamedTuple):
budget: str
rule: str
detail: str


def _run(cmd: list[str]) -> subprocess.CompletedProcess[str]:
return subprocess.run(cmd, cwd=REPO_ROOT, capture_output=True, text=True)


def _merge_base(base: str) -> str:
"""The common ancestor of `base` and HEAD, so unrelated base drift is ignored."""
proc = _run(["git", "merge-base", base, "HEAD"])
return proc.stdout.strip() or base


def _load_head(rel: str) -> dict | None:
path = REPO_ROOT / rel
if not path.exists():
return None
return json.loads(path.read_text())


def _load_base(rel: str, ref: str) -> dict | None:
"""Budget content at `ref`, or None when the file did not exist there."""
proc = _run(["git", "show", f"{ref}:{rel}"])
if proc.returncode != 0:
return None
return json.loads(proc.stdout)
Comment thread
greptile-apps[bot] marked this conversation as resolved.
Resolved


def _caps(budget: dict) -> dict[str, int]:
"""Map each rule to its ceiling (baseline + slack); skip malformed specs."""
caps: dict[str, int] = {}
for rule, spec in budget.items():
if isinstance(spec, dict):
caps[rule] = int(spec.get("baseline", 0)) + int(spec.get("slack", 0))
return caps


def regressions_for(rel: str, base: dict | None, head: dict | None) -> list[Regression]:
if base is None:
return [] # new budget file: nothing to ratchet against yet
if head is None:
return [Regression(rel, "*", "budget file was deleted (every ceiling removed)")]

base_caps = _caps(base)
head_caps = _caps(head)
out: list[Regression] = []
for rule, base_cap in sorted(base_caps.items()):
if rule not in head_caps:
out.append(Regression(rel, rule, f"rule dropped (ceiling {base_cap} -> removed)"))
elif head_caps[rule] > base_cap:
out.append(Regression(rel, rule, f"ceiling raised {base_cap} -> {head_caps[rule]}"))
return out


def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--base", default=DEFAULT_BASE)
parser.add_argument("budgets", nargs="*", help="budget files to check")
args = parser.parse_args()
budgets = args.budgets or list(DEFAULT_BUDGETS)

ref = _merge_base(args.base)
regressions: list[Regression] = []
checked: list[str] = []
for rel in budgets:
base = _load_base(rel, ref)
head = _load_head(rel)
if base is None and head is None:
continue
if base is None:
print(f"skip {rel}: new file (no base at {args.base} to ratchet against)")
continue
checked.append(rel)
regressions.extend(regressions_for(rel, base, head))

if regressions:
print(f"FAIL: budget ceiling(s) loosened vs base {args.base} (merge-base {ref[:12]}):")
for reg in regressions:
print(f" {reg.budget} {reg.rule}: {reg.detail}")
print(
"Budgets are one-way ratchets and may only go down or stay flat. This "
"check is non-gating: if the increase is justified (e.g. a newly banned "
"API), a human can merge over the red after acknowledging it."
)
return 1

suffix = f" ({', '.join(checked)})" if checked else ""
print(f"OK: no budget ceiling increased vs base {args.base}{suffix}")
return 0


if __name__ == "__main__":
raise SystemExit(main())
Loading
Loading