ci: gate PR bodies on the template's HTML comment rules

[mateo-berri]

TLDR

Problem this solves:

• The PR template's rules hide inside HTML comments
• Rendered PRs and agent harnesses both drop those comments
• Authors, human and AI, routinely miss the rules

How it solves it:

• New CI job checks PR bodies against the comment rules
• Enforces short bullets, no placeholders, no stray QA runbook
• Re-runs on every body edit and push

User Flow

Before: a contributor writes the Caveats section as prose and leaves the QA runbook in, and nothing tells them until a maintainer reads it

1. They open a PR at https://github.com/BerriAI/litellm/compare with a body whose Caveats section is a paragraph and whose QA runbook survived from the template
2. Every check on https://github.com/BerriAI/litellm/pull/{number}/checks turns green
3. Days later a maintainer reviewing the PR asks them to reformat the description by hand

After: the same PR gets an immediate failing check naming each violated rule

1. They open a PR at https://github.com/BerriAI/litellm/compare with a body whose Caveats section is a paragraph and whose QA runbook survived from the template
2. The "PR Body Template / Validate PR body" check on https://github.com/BerriAI/litellm/pull/{number}/checks fails within a minute, its log naming the prose line and the stray QA runbook section
3. They fix the description in place, the check re-runs on the edit event and turns green

Relevant issues

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Both runs captured at 5d9f28c against PR #36543's real description and changed-file list. The before input is that PR's description exactly as it stood before its bullets fix, which a reviewer had to catch by hand; the after input is the description live right now

$ gh pr view 36543 --json body --jq .body > body_current.txt
$ gh api "repos/BerriAI/litellm/pulls/36543/files" --paginate --jq '.[].filename' > files.txt

$ PR_BODY="$(cat body_before_bullets_fix.txt)" python3 .github/scripts/check_pr_body.py --changed-files files.txt
::error title=PR body template::Caveats (if any): prose line "Three micro-hardenings ride along with the typing because th" must be a short bullet instead
::error title=PR body template::QA runbook: delete this section; the template only wants it when the PR edits tests/e2e

The rules above come from the HTML comments inside .github/pull_request_template.md; open that file to see every rule next to its section.
$ echo $?
1

$ PR_BODY="$(cat body_current.txt)" python3 .github/scripts/check_pr_body.py --changed-files files.txt
PR body follows the template comment rules.
$ echo $?
0

On this PR itself the check skips with a notice, since the judging copy always comes from the base branch and the base branch gains the script only when this merges; the captured runs above are the proof

Type

🚄 Infrastructure

Caveats (if any)

• Hard cap is 14 words to keep roughly-10 workable
• Skippable per PR with the ignore-pr-body-template label
• Bot-authored PRs are skipped entirely
• Only the machine-checkable comment rules are enforced
• Fenced code blocks are ignored when parsing sections and placeholders
• Checker always runs from the base branch, never the PR's copy
• Skips with a notice while the base branch lacks the script
• Startup-safety checker now exempts object-filter dereferences, not arithmetic
• Making the check required is a branch protection follow-up

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

---

Note

Medium Risk
New CI gate that runs on every human PR and can fail on description formatting; false positives would block contributors until they reformat or apply the skip label. No production runtime impact.

Overview
Adds a CI check that enforces the machine-checkable rules buried in HTML comments in pull_request_template.md, which rendered PRs and agent harnesses normally strip away.

The new workflow validates that TLDR and Caveats use short bullets (≤14 words), leftover placeholders like <blah>/... are gone, and the QA runbook section is removed unless the PR touches tests/e2e/. It re-runs on body edits, skips bots and the ignore-pr-body-template label, and always judges from the base branch copy of the script so a PR cannot weaken the gate.

Also exempts GitHub's labels.*.name object-filter syntax from the workflow startup-safety arithmetic checker, which the new workflow needs.

^{Reviewed by Cursor Bugbot for commit e02c331. Bugbot is set up for automated code reviews on this repo. Configure here.}

[greptile-apps]

Greptile Summary

The PR adds a CI gate that validates pull request bodies using a checker sourced exclusively from the base revision.

• Validates bullet formatting, placeholders, and conditional QA runbook removal.
• Re-runs when pull request content, labels, or commits change.
• Removes the previously reported PR-controlled checker and fallback paths.
• Adds checker and workflow-safety test coverage.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains; the validator now comes solely from the base revision and the PR-controlled fallback has been removed.

Important Files Changed

Filename	Overview
.github/workflows/pr-body-template.yml	Checks out and executes only the base revision’s validator, fully addressing both previously reported trust-boundary bypasses.
.github/scripts/check_pr_body.py	Implements deterministic pull request body validation for the template’s machine-checkable rules.
tests/test_litellm/test_github_check_pr_body.py	Covers compliant bodies, formatting violations, placeholders, fenced content, HTML comments, and QA runbook handling.
tests/code_coverage_tests/check_workflow_startup_safety.py	Exempts GitHub Actions object-filter syntax from multiplication detection without weakening arithmetic checks.

_{Reviews (5): Last reviewed commit: "fix: ignore fenced code blocks when chec..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

Autofix Details

Bugbot Autofix prepared fixes for both issues found in the latest run.

• ✅ Fixed: Ellipsis false positives everywhere
  • Restricted the ... placeholder match to bullet lines (- ...), which is the only form the template uses, so bare ... in prose or omitted output no longer trips the check.
• ✅ Fixed: Fenced headings break parsing
  • Made split_sections track fenced code-block state via a scan and ignore heading-looking lines that appear inside fences so quoted template snippets no longer conjure fake sections.

Or push these changes by commenting:

@cursor push 0f42d30c34

Preview (0f42d30c34)

diff --git a/.github/scripts/check_pr_body.py b/.github/scripts/check_pr_body.py
--- a/.github/scripts/check_pr_body.py
+++ b/.github/scripts/check_pr_body.py
@@ -25,6 +25,7 @@
 HEADING_PATTERN: Final = re.compile(r"^#{2,6}\s+(?P<title>.+?)\s*$")
 BULLET_PATTERN: Final = re.compile(r"^\s*(?:[-*+]|\d+[.)])\s+(?P<text>.*)$")
 LABEL_PATTERN: Final = re.compile(r"^[^-*+].*:\s*$")
+FENCE_PATTERN: Final = re.compile(r"^\s*(?:```|~~~)")
 
 
 @dataclass(frozen=True, slots=True)
@@ -48,9 +49,21 @@
 
 def split_sections(body: str) -> tuple[Section, ...]:
     lines: Final = tuple(body.split("\n"))
-    headings: Final = tuple(
-        (index, match.group("title")) for index, line in enumerate(lines) if (match := HEADING_PATTERN.match(line))
-    )
+
+    def scan(
+        acc: tuple[bool, tuple[tuple[int, str], ...]],
+        item: tuple[int, str],
+    ) -> tuple[bool, tuple[tuple[int, str], ...]]:
+        in_fence, headings = acc
+        index, line = item
+        if FENCE_PATTERN.match(line):
+            return (not in_fence, headings)
+        if in_fence:
+            return acc
+        match: Final = HEADING_PATTERN.match(line)
+        return (in_fence, (*headings, (index, match.group("title")))) if match else acc
+
+    _, headings = reduce(scan, tuple(enumerate(lines)), (False, ()))
     ends: Final = tuple(index for index, _ in headings[1:]) + (len(lines),)
     return tuple(Section(title=title, lines=lines[start + 1 : end]) for (start, title), end in zip(headings, ends))
 
@@ -92,8 +105,9 @@
 
 def is_placeholder_line(line: str) -> bool:
     bullet_match: Final = BULLET_PATTERN.match(line)
-    content: Final = bullet_match.group("text").strip() if bullet_match else line.strip()
-    return content == "..." or any(token in line for token in PLACEHOLDER_TOKENS)
+    if bullet_match and bullet_match.group("text").strip() == "...":
+        return True
+    return any(token in line for token in PLACEHOLDER_TOKENS)
 
 
 def check_placeholders(sections: tuple[Section, ...]) -> tuple[Violation, ...]:

diff --git a/tests/test_litellm/test_github_check_pr_body.py b/tests/test_litellm/test_github_check_pr_body.py
--- a/tests/test_litellm/test_github_check_pr_body.py
+++ b/tests/test_litellm/test_github_check_pr_body.py
@@ -109,3 +109,28 @@
     violations = checker.check_body(body, ("litellm/main.py",))
     assert len(violations) == 1
     assert violations[0].section == "QA runbook"
+
+
+def test_bare_ellipsis_in_proof_output_is_not_a_placeholder(checker):
+    body = (
+        "## Screenshots / Proof of Fix\n\n"
+        "```\n"
+        "$ curl http://localhost:4000/v1/chat/completions ...\n"
+        "{\"id\": \"chatcmpl-abc\",\n"
+        " ...\n"
+        " \"usage\": {\"prompt_tokens\": 5}}\n"
+        "```\n"
+    )
+    assert checker.check_body(body, ()) == ()
+
+
+def test_headings_inside_fenced_block_do_not_open_new_sections(checker):
+    body = (
+        "## Screenshots / Proof of Fix\n\n"
+        "Quoted template excerpt:\n\n"
+        "```markdown\n"
+        "## Caveats (if any)\n\n"
+        "Some prose paragraph that is not a bullet at all\n"
+        "```\n"
+    )
+    assert checker.check_body(body, ()) == ()

_{You can send follow-ups to the cloud agent here.}

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit e02c331. Configure here.}