[eeeeeio]

Summary

• Add skills/npu-perf-optimization/ — a systematic performance optimization loop for SGLang on Ascend NPU: reproducible baseline collection, operator-level bottleneck attribution, a quantified A/B matrix over NPU feature switches (NPU Graph, DeepEP, DP-Attention, MTP/speculative), and a structured report with a recommended launch command.
• Complements the existing sglang-npu-adapter and npu-api-adaptation-checker skills in this repo; this skill adds the orchestration layer: matrix planning, JSON contracts, comparability rules, and recommendation synthesis.

Skill functionality

When to invoke: a model already serves on Ascend NPU and you need a defensible optimal config or before/after numbers per switch; or a deployment regressed and you need to locate the config regression. Not intended for NVIDIA/CUDA deployments or for single one-off trace analysis.

The loop — SKILL.md orchestrates five phases; all handoff is via JSON files on disk, scripts are idempotent and need no interactive session:

Phase	Script	What it does	Output
0. Preflight	mandatory checks in SKILL.md	environment/toolchain, proxy variables unset, hardware topology (A3 dual-die → --tp-size = 2 × card count), model weights (draft weights only for draft-based speculative options), workdir layout (traces never written to /tmp); fail fast until all pass	gate
1. Baseline	collect_baseline.py	launch from the closest official best-practice config; sanity guard (20 short generations); frozen workload spec run 3 times, median kept	baseline/baseline.json
2. Bottleneck attribution	analyze_profile.py	benchmark + trace in one run; parses operator/step tables; top-N operators, prefill/decode split, attribution ∈ {compute, comm, launch, mixed}	bottleneck.json
3. A/B matrix	run_ab_matrix.py	single-variable P0 pass over NPU Graph (ACLGraph) / DeepEP / DP-Attention / MTP-speculative; compatibility gate rejects illegal combos before GPU time; 3% noise floor; per-variant circuit breaker; resumable checkpointing	ab/*.json
4. Report	gen_report.py	mandatory per-switch before/after table (TPOT, TTFT, throughput); bottleneck-to-switch mapping; full copy-pasteable recommended launch command with expected deltas and one-line rationale per switch; provenance (hardware, sglang commit, workload spec, run dates)	reports/report.md + report.json

Quality gates built into the skill: every number in the report must trace to a stored JSON artifact; 3-run median everywhere; no improvement is claimed below the 3% noise floor; the recommended config must pass the --validate-only compatibility check; the workload spec is stated verbatim for reproducibility.

Mapping to the issue's acceptance criteria (Ascend/sglang#1109):

• "SKILL.md: Fully self-consistent with trigger timing, mandatory checks, input/output JSON contract, and write paths for the new phase" → SKILL.md + references/json-schemas.md
• "Prompt and invocation templates for the new sub-agent are complete" → references/subagent-contracts.md (machine contracts for three sub-agents: baseline-collector, bottleneck-analyzer, matrix-runner)
• "New scripts can collect performance baselines and generate machine-readable JSON + optimization reports, with reports including bottleneck identification results and quantified before/after comparisons for each feature switch (ACLGraph / DeepEP / DP-Attention / MTP)" → the A/B matrix ("NPU Graph" in the skill's naming = ACLGraph) driven by collect_baseline.py / analyze_profile.py / run_ab_matrix.py / gen_report.py, with the schemas and report template in references/
• "Produce a set of directly reusable 'Recommended Launch Commands' and quantify the performance improvement relative to the baseline" → the recommended-command section of the generated report.md (template in references/report-template.md), backed by the hardware numbers in the Test plan below

Task

Ascend/sglang#1109

The skill deliverable (SKILL.md + scripts + references) was included in the Ascend Task Center acceptance package submitted on 2026-09-24 (under review there); this PR adds it to the skills repo as guided in the task discussion group, with minor parsing/validation fixes applied since the package submission.

Deliverables

• SKILL.md (trigger timing, mandatory checks, JSON contracts, write paths)
• scripts/: collect_baseline.py, analyze_profile.py, run_ab_matrix.py, gen_report.py, smoke_test.py
• references/: json-schemas.md, feature-switch-matrix.md, subagent-contracts.md, report-template.md

Test plan

• python scripts/smoke_test.py — 19/19 checks pass
• Full loop on hardware: Atlas A2 (Qwen3-8B bf16) — baseline TPOT 14.77 ms, bottleneck attribution (MatMul 83.98% of kernel time), ACLGraph A/B (graph off: +205.48% worse); Atlas A3 dual-die (Qwen3-0.6B; Qwen3-30B-A3B MoE for DeepEP) — NGRAM (−51% TPOT), DP-Attention 10.53 ms, DeepEP 30.23 ms verified. Full evidence accompanies the task's acceptance submission.