[Docs] Add PR checklist skill for coding agents

[benchislett]

Overview

(DISCLAIMER: I USED THE NEW SKILL TO HAVE GPT-6 AUTHOR THE PR DESCRIPTION BELOW, COMMIT THE CHANGES AND OPEN THIS PR. TREAT IT AS A REVIEWABLE EXAMPLE ARTIFACT. ALL CODE CHANGES WERE HAND WRITTEN WITH CARE BY ME.)

Add a pr-checklist skill for coding agents preparing, reviewing, or revising
vLLM pull requests. Integrate it with the agent contribution instructions and
the PR template.

Claims

• Provide a structured review workflow covering design fit, compatibility,
  complexity, tradeoffs, validation, test quality, code clarity, and PR evidence.
• Distinguish validation performed for a contribution from tests maintained in
  the repository, emphasizing meaningful coverage rather than test volume.
• Include guidance for human-in-the-loop review and responding to prior feedback.
• Require agents to use the skill before submission through AGENTS.md.
• Restructure the PR template around Overview, Claims, Validation, and Details,
  with checklist and AI-assistance disclosure fields.

The intended benefit is better-prepared contributions and less avoidable
maintainer follow-up. This PR does not claim a measured reduction in review
rounds or review time.

Validation

This change is documentation and agent instructions only. No serving-engine,
model, kernel, dependency, or runtime behavior is modified; runtime tests,
model evaluations, and performance benchmarks are not applicable.

Diff and structure checks

git status --short
git diff -- AGENTS.md .github/PULL_REQUEST_TEMPLATE.md
git ls-files --others --exclude-standard .agents/skills/pr-checklist
git diff --check
git diff --no-index --check /dev/null .agents/skills/pr-checklist/SKILL.md
wc -l AGENTS.md .agents/skills/pr-checklist/SKILL.md

• Reviewed the two tracked-file diffs and the complete new skill.
• Confirmed the proposed change consists of those three files; research data,
  scraping scripts, plots, and intermediate drafts are excluded.
• AGENTS.md has 159 lines and the skill has 153 lines, within the applicable
  instruction-document budgets.
• The tracked-file whitespace check produced no diagnostics.
• The new skill's whitespace check reports one trailing space on the helper
  inlining instruction. This remains in the proposed diff.
• No Markdown linter or documentation build was run.

Current-skill behavioral smoke test

A fresh agent context read the current skill and applied it to four
hypothetical scenarios without executing tests or changing files:

1. This documentation-only PR: recognized runtime validation as not
   applicable and assessed workflow consistency and documentation.
2. Reviewing another author's optimization PR: requested missing
   configuration, correctness, and serving-performance evidence rather than
   skipping the PR-contents section.
3. Re-review with an unresolved backend concern: traced related paths,
   requested appropriate evidence, and identified a stale description.
4. Intentional all-drafts-rejected regression test: identified a conflict
   between the expected zero acceptance and the skill's nonzero-acceptance
   instruction. This remains an outstanding caveat below.

These are qualitative instruction-following checks, not automated regression
tests or evidence of real-world review-time savings.

Details

Motivation and research

The drafting process examined the latest 50 vLLM PRs reviewed by each of
benchislett and mgoin, ordered by the reviewer's latest submitted review.
The samples contain 97 distinct PRs and 342 nonempty authored review/comment
messages. Recurring themes included compatibility across execution paths,
appropriate ownership and reuse, concise implementations, meaningful rather
than excessive tests, and claims supported by evidence.

A separate exploratory analysis measured additions and deletions under
vllm/ and compared them with description-based complexity grades. It
reinforced that LOC is a coarse review aid, not a reliable standalone semantic
complexity classification. The skill's 50/250 bands are judgment-based
guidance, not validated predictive cutoffs.

The research artifacts are local preparation material and are not included
in this PR. The proposed skill was handwritten and iteratively refined by
the human author with AI-assisted research and review.

Duplicate-work checks

Open-PR searches were run on September 15, 2026:

gh pr list --repo vllm-project/vllm --state open \
  --search '"pr-checklist"' --limit 30 --json number,title,url,headRefName
gh pr list --repo vllm-project/vllm --state open \
  --search '"skill" "review"' --limit 30 --json number,title,url,headRefName
gh pr list --repo vllm-project/vllm --state open \
  --search 'checklist in:title' --limit 30 --json number,title,url
gh pr list --repo vllm-project/vllm --state open \
  --search 'skill in:title' --limit 30 --json number,title,url
gh pr list --repo vllm-project/vllm --state open \
  --head 'benchislett:skill-pr-checklist' --json number,title,url

The broad searches returned unrelated PRs mentioning review or checklist
usage. Targeted title searches found checklists for tool-calling models and
server hardening, and skills for CI and installation; none appeared to
duplicate this general contribution-review workflow. No existing open PR
was found for this branch. No associated issue was supplied, so issue-specific
duplicate checks were not applicable.

AI assistance

AI assistance was used for GitHub review-history collection, qualitative
categorization, exploratory LOC analysis, draft review, behavioral smoke
checks, and preparation of this description. The human submitter remains
responsible for reviewing and understanding every changed line.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill.
• AI assistance was used during the creation of this PR.
• Design Fit: Adds a focused skill and integrates it into the contribution workflow without runtime changes.
• Testing and Validation: Documentation checks and behavioral smoke cases were run; the rejection-test instruction caveat and whitespace diagnostic remain outstanding.
• Code Quality and Style: Review the helper instruction and invocation/disclosure wording noted above; no lint pass has been claimed.
• Pull Request Contents: Describes the purpose, scope, evidence, duplicate-work checks, limitations, and AI assistance.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[mergify]

Documentation preview: https://vllm--57084.org.readthedocs.build/en/57084/

[sfeng33]

These might be better fitted in each folder's agent.md, as different part of the codebase has different guidelines.

[benchislett]

Good point, I agree

[faradawn]

Love the statement that it is "mandatory for agents, optional for humans".

Some reviewer pointed out that agent generated PR descriptions tend to be long. Do we want to "limit to 100-200 words"?

LGTM overall

[benchislett]

@faradawn I indeed share the frustration of overly wordy PR descriptions. However, we don't want to suppress agents from sharing relevant drawbacks.

My compromise here was to enforce header sections that are length-constrained:

Ensure the PR description has a short 1-2 line summary of the changes to facilitate quick understanding by reviewers

Perhaps I should extend this to explicitly mention the "claims" section.

For Validation and Notes sections, I thought it would be better to let these sections be wordy -- we often do want to see extensive validation results, and a detailed root-cause trace, for tricky issues.

I thought about requiring that these be sorted into details:

example

some very long validation content could go in here

But I'm not sure if that adds much value if there's not anything else in the description we're trying to scroll towards.

[faradawn]

Good point that we want the agents to share potential drawbacks. I like the idea that of a "short 1-2 line summary".

[faradawn]

Tagging survey respondents who flagged this. Please react 👍 or comment.

@AndreasKaratzas — you flagged that agent PR have long text. Does the "1–2 line limit" fix that?

@tlrmchlsmth — "way too much text everywhere." Does this PR address your concern?

Also @njhill, @bbrowning, @DarkLight1337, @tjtanaa, @mgoin, @esmeetu, @aoshen02, @penli9 - 👍 or comments welcome.

Aiming to merge by Friday Sep 25.

[AndreasKaratzas]

My prompt to the agent usually is "write 3-5 bullet point list with your modifications followed up by a 3-6 sentence mini-paragraph with the story and motivation behind these changes" (including buildkite link or regression PR if the PR is targeting a CI regression fix). For kernel PRs, I do like the occasional print screen of the traces as well, to ensure that this was not 100% vibe coded, contrary it was validated and the author rendered the traces before and after their modifications. Also table about gains and perf evals are good, but again a prompt could be limit this to one lm eval table and a bench table that contains average and stats across 10 bench runs.

[njhill]

Thanks @benchislett for taking to the initiative with this. Looks great to me overall. I've suggested a few additions inline.

Something else we could consider adding but possibly as a later improvement, is pointers to examples of some of the points, in the form of prior PR or existing code sections / features that exemplify our expectations.

For Validation and Notes sections, I thought it would be better to let these sections be wordy -- we often do want to see extensive validation results, and a detailed root-cause trace, for tricky issues.

I think it would still be worthwhile to emphasize conciseness for explanations etc. I often find it quite time consuming to wade through verbose slop descriptions to understand the salient points.

[njhill]

@benchislett just in case it wasn't obvious, most of my comments are suggesting additions rather than replacements.

Also I just had another thought that we could add a specific directive to ensure cpu<->gpu checks aren't introduced on the main model execution thread (though we now have the checker in CI, some still slip through since not all configurations are exercised in CI).

[benchislett]

Got very busy this weekend. Will update tomorrow. Thanks very much everyone for feedback!

[benchislett]

Going to trigger CI just to see what it wants to run. Ideally nothing, since there are no code changes

[benchislett]

/ci run

[github-actions]

✅ Triggered Buildkite CI #92177 for commit a50019c97914.

[qiching]

LGTM now

[vllm-agent]

CI selector (shadow): 0 test steps (0 jobs) instead of 0 (0 jobs)

Shadow mode: this changes nothing about what CI runs. It shows what the evidence-based selector would pick for this PR, next to today's rules. How it works.

Feedback welcome: reply here if it would skip a step this change needs, or runs something unrelated.

steps (jobs)	Today's rules	Selector	Would skip	Would add
NVIDIA, CPU and others	0 (0)	0 (0)	0 (0)	0 (0)
AMD mirrors	0 (0)	0 (0)	0 (0)	0 (0)

Docs-only change: today's rules run no tests.

Selector would run (0)

none

Would skip (today's rules run them) (0)

none

Would add (today's rules do not run them) (0)

none

AMD mirrors: would skip (0)

none

AMD mirrors: would add (0)

none

CI results (2026-09-30 18:52 UTC)

0 passed, 0 failed, 0 pending.

No failures to judge.

_{4 changed files · base 2df122e65e · head a50019c979 · Python record: not used (/tmp/ci-infra-selector/buildkite/ci_selector/coverage-data/table.json.gz is table version 5, expected 7; re-merge it from the raw recordings) · kernel record: table 866fa13 (build 92059), map 866fa13 · not counted: 10 build steps, 5 A100 steps the generator no longer emits, 0 optional steps the selector would also run}

[njhill]

Thanks again @benchislett!