[Skill] add perf-bisect for attributing vllm-omni perf regressions to commits

[linyueqian]

What

Adds .claude/skills/perf-bisect/ — a project-local Claude Code skill that gives vllm-omni contributors a repeatable workflow for attributing a perf change (TTFP, RTF, audio throughput, image latency, step time) to a specific commit. Generalised from the TTS-only workflow used during the post-#3662 regression hunt (#3681 / #3817 / #3839) so the same discipline applies to diffusion-image and omni-audio bisects.

Why

PR #3839's perf regression took several hours to attribute, mostly because of one trap: an initial bisect measured the wrong cell (Base voice_clone with the default deploy YAML) when the report was about CustomVoice default_voice under qwen3_tts_high_concurrency.yaml. The bench completed cleanly with apples-vs-oranges numbers and reported "no regression" while a +118% TTFP regression was live. This skill encodes the cell-definition discipline — extract all seven dimensions (model, task, deploy_yaml, dataset, num_prompts, max_concurrency, num_warmups) plus family-specific knobs from the regression report before writing any bench script — so the trap is harder to fall into next time.

Layout

.claude/skills/perf-bisect/
├── SKILL.md                                  # discipline + 5-step workflow
├── references/
│   ├── family-knobs.md                       # extra_body / stage_overrides per family
│   └── pitfalls.md                           # 6 mechanical failure modes + remediations
└── scripts/
    ├── run_bisect.sh                         # bench-loop template
    ├── kanban_trend.py                       # metric time series from vllm-omni-kanban
    └── cells/
        ├── README.md                         # <family>_<descriptor>.yaml convention
        ├── tts_default_voice_high_c.yaml     # #3839's regression cell
        └── tts_voice_clone_nightly.yaml      # kanban-parity cell

Contents

• SKILL.md — trigger phrases (English + Chinese), paired tools (remote-gpu, tts-perf-check, the kanban repo), the generic 7-tuple cell-discipline table, a family-specific knob TL;DR, the 5-step workflow (kanban triage → bisect span → bench harness → interpret → variance check), a rationalization table of excuses-vs-reality, and a red-flags pre-flight list.
• references/family-knobs.md — full knob tables for TTS / diffusion-image / omni-audio (task_type, voice, language, width, height, num_inference_steps, stage_overrides) plus the headline metric each family reports.
• references/pitfalls.md — six failure modes caught in real bisects, each with a copy-paste remediation snippet: pytest -k zero-match, venv PATH not inherited by subprocess, stale server PID binding the wrong port, multi-tenant GPU contention, /v1/models returning before CUDA-graph compile, cold model download exceeding the server-ready timeout.
• scripts/run_bisect.sh — bench-loop template that pairs vllm serve (with --deploy-config) and vllm bench serve --omni, polls /v1/models with a 30s settle window, parses median/p99 TTFP + RTF + throughput from the saved JSON, and cleans up the server between commits.
• scripts/kanban_trend.py — prints a per-build metric time series from vllm-omni-kanban with rolling-delta percent and ←REG / ←IMP markers at the 10% threshold; works for any cell prefix the kanban tracks.
• scripts/cells/ — two production cells (the [Perf][TTS] Restore Qwen3-TTS default_voice c=64 TTFP to v021 baseline #3839 high-c regression cell and the Base voice_clone kanban-parity cell) plus a README documenting the <family>_<descriptor>.yaml convention so diffusion_* / omni_* cells can be added without collisions.

Triggers

Natural-language phrases that should activate the skill:

• "find which PR regressed Qwen3-TTS perf"
• "bisect TTFP between commit X and Y"
• "verify PR #N actually improves perf"
• "高并发 TTFP 劣化"

Scope

This skill is investigation, not prevention. Writing a new perf regression test belongs in tests/dfx/perf/. Reading existing perf trends belongs in vllm-omni-kanban. Cross-version (vllm 0.20 ↔ 0.21) bisects need two venvs and are explicitly out of scope.

Verification

• Skill picked up by Claude Code's skill registry (verified locally — Skill(perf-bisect) invokes correctly).
• All five resource references in SKILL.md resolve to files in the diff.
• No session-detail (dates, host names, reproduction logs) in any docstring or comment.
• Quality rubric: 96/100 (A) — description 97, organization 94, style 92, structure 100.

How to use

In any Claude Code session inside this repo:

@perf-bisect bisect TTFP between abc123 and def456 for CustomVoice default_voice c=64 N=512 with the high-c YAML

The skill will ask for the missing cell dimensions if any are unspecified, then walk through kanban triage → bisect span → bench loop → interpretation.

Out of scope for this PR

• Other model families' cell files (anyone can add diffusion_*.yaml / omni_*.yaml in a follow-up).
• Wiring this into Buildkite as an automated CI step. A separate PR will propose a tts-perf-regression-ci mechanism that uses this skill's cell convention for its specs.

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[linyueqian]

Closing this — on second look the skill content is too tied to one team's internal regression hunt to live in the public repo. Planning to contribute it to a more appropriate home (hsliu's perf-tracking repo) instead. Sorry for the noise.