feat: add npu-perf-optimization skill for SGLang Ascend NPU performance tuning

skills/npu-perf-optimization/SKILL.md

@@ -0,0 +1,137 @@
+---
+name: npu-perf-optimization
+description: Systematic performance optimization loop for SGLang on Ascend NPU — collect a reproducible baseline, identify operator-level bottlenecks, run a quantified A/B matrix over NPU feature switches (NPU Graph, DeepEP, DP-Attention, MTP/speculative), and emit a structured report with a recommended launch command. Invoke when a model already serves on Ascend NPU and you need defensible optimal configs or before/after numbers per switch, or when a deployment regressed and you need to find the config regression. Run with /npu-perf-optimization.
+---
+
+# NPU Performance Optimization Loop (Ascend)
+
+Turns "the model runs, now make it fast" into a reproducible pipeline:
+**baseline → bottleneck attribution → feature-switch A/B matrix → report + recommended launch command**.
+
+Complements existing skills — do not duplicate them:
+- model/feature adaptation is the prerequisite: `sglang-npu-adapter`
+      (model adaptation) and `npu-api-adaptation-checker` (API-parameter
+      adaptation status), both in the Ascend-SGLang-Skills repo
+- trace capture patterns come from `generate-profile` and trace analysis from
+  `llm-torch-profiler-analysis` (both shipped in sgl-project/sglang under
+  `.agents/skills/`)
+- this skill adds the **orchestration layer**: matrix planning, JSON contracts,
+  comparability rules, and recommendation synthesis, specialized for Ascend NPU.
+
+## When to run
+
+- After a model is adapted and serving on Ascend NPU and you need a defensible
+  "optimal config" rather than folklore tuning.
+- When a deployment regressed and you need before/after numbers per switch.
+- NOT for: CUDA/NVIDIA deployments (use upstream tuning guides), or when only a
+  single trace analysis is needed (use `llm-torch-profiler-analysis` directly).
+
+## Hard prerequisites (Phase 0 — mandatory checks)
+
+Refuse to proceed (fail fast with a clear message) until every box passes:
+
+- [ ] `source /usr/local/Ascend/ascend-toolkit/set_env.sh` and
+      `source /usr/local/Ascend/nnal/atb/set_env.sh` succeeded
+- [ ] `npu-smi info` shows the expected hardware; **A3 = 2 dies per card →
+      `--tp-size` is 2 × card count**; record `hardware` into every JSON meta
+- [ ] model weights exist; if speculative decoding with a draft model is
+      planned (e.g. EAGLE3), draft-model weights exist too (training-free
+      options like NGRAM need none)
+- [ ] proxies unset: `unset http_proxy https_proxy HTTP_PROXY HTTPS_PROXY`;
+      `unset ASCEND_LAUNCH_BLOCKING`
+- [ ] workdir layout created and writable:
+      `<workdir>/{baseline,ab,profiles,reports}/` (never write traces to `/tmp`)
+- [ ] output JSON paths planned; schema = `references/json-schemas.md`
+
+## Workflow
+
+### Phase 1 — Baseline (`scripts/collect_baseline.py`)
+
+1. Launch the server from the closest official best-practice config
+   (`docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/`).
+2. Wait for readiness: poll `/health`, and grep server log for
+   "The server is fired up and ready to roll!".
+3. Sanity guard: 20 short generations; abort baseline if any fail.
+4. Run `python -m sglang.bench_serving` with a **frozen workload spec**
+   (dataset, lens, concurrency, seed) **3 times**; keep the median.
+   Comparability rule: every later A/B variant MUST reuse the identical spec.
+5. Emit `baseline/baseline.json` (schema in `references/json-schemas.md`).
+
+### Phase 2 — Bottleneck attribution (`scripts/analyze_profile.py`)
+
+1. `bench_serving --profile --profile-steps 10 --profile-output-dir profiles/`
+   (Method C: benchmark + trace in one command; server auto-parses).
+2. Read `operator_details.csv` / `step_trace_time.csv` from the parsed dir
+   (path printed as `Profiling done. Traces are saved to: <path>`).
+3. Emit `bottleneck.json`: top-N operators by share, prefill/decode split,
+   attribution ∈ {compute, comm, launch, mixed}.
+4. Note: full Python-stack visibility requires `--disable-cuda-graph` at launch
+   (perf-penalizing — profiling runs only). To profile graph capture itself use
+   `--enable-profile-cuda-graph`.
+
+### Phase 3 — Feature-switch A/B matrix (`scripts/run_ab_matrix.py`)
+
+Matrix definition lives in `references/feature-switch-matrix.md`. Summary:
+
+| Switch | Knobs | Models |
+|---|---|---|
+| NPU Graph | `--cuda-graph-bs` list / `--disable-cuda-graph` / `--cuda-graph-max-bs-decode` | all |
+| DeepEP | `--moe-a2a-backend deepep`, `--deepep-mode`, dispatch-tokens env, int8 quant env | MoE only |
+| DP-Attention | `--enable-dp-attention`, `--dp-size`, `--enable-dp-lm-head`, skip-all-gather env | TP ≥ 2 |
+| MTP / speculative | `--speculative-algorithm` (NEXTN for DeepSeek, EAGLE3 for Qwen, NGRAM training-free), steps/topk/draft-tokens | per model |
+
+Rules (enforced by the script, not by memory):
+- **Single-variable first**: P0 pass toggles each switch on/off vs baseline;
+  P1 pass explores interactions only for switches that moved >3%.
+- **Compatibility gate**: reject combos violating the official feature matrix
+  (e.g. torch-compile with NPU Graph; EPLB with `ascend_fuseep`;
+  `TASK_QUEUE_ENABLE=2` with graph capture) before spending GPU time.
+- **Effect size**: |ΔTPOT or Δthroughput| ≤ 3% vs baseline ⇒ report as "no
+  significant change"; never word it as an improvement.
+- **Circuit breaker per variant**: server fails to start, sanity fails, or
+  completed < expected prompts ⇒ record `status: failed` with the log tail and
+  continue to the next variant; never abort the whole matrix.
+- **Checkpointing**: matrix state file allows resume after interruption; each
+  variant's raw bench JSON is kept.
+
+### Phase 4 — Report (`scripts/gen_report.py`)
+
+Emits `reports/report.md` + `reports/report.json` from
+baseline/bottleneck/ab artifacts:
+
+- per-switch before/after table (TPOT, TTFT, throughput) — **mandatory section**
+- bottleneck summary and which switch addresses it
+- **Recommended launch command**: the full, copy-pasteable command (env exports
+  + `python -m sglang.launch_server ...`) with expected deltas vs baseline,
+  and one-line rationale per switch
+- provenance: hardware, sglang commit, workload spec, run dates
+
+## Quality gates (before declaring done)
+
+- [ ] every number in the report traces to a stored JSON artifact (no
+      unsourced numbers)
+- [ ] recommended config passes the compatibility matrix check in
+      `run_ab_matrix.py --validate-only`
+- [ ] 3-run median used everywhere; raw runs included
+- [ ] report states workload spec verbatim (reproducibility)
+- [ ] no claim of improvement below the 3% noise floor
+
+## Sub-agent contracts
+
+This skill is designed to be driven by coding agents. Machine contracts
+(prompts, invocation templates, timeouts, failure semantics) for the three
+sub-agents (`baseline-collector`, `bottleneck-analyzer`, `matrix-runner`) are in
+[references/subagent-contracts.md](references/subagent-contracts.md). All
+inter-agent handoff is via JSON files on disk — scripts are idempotent and do
+not require an interactive session.
+
+## Deep references (read on demand, don't inline)
+
+- [references/feature-switch-matrix.md](references/feature-switch-matrix.md) —
+  full parameter space per switch + compatibility constraints
+- [references/json-schemas.md](references/json-schemas.md) —
+  baseline / ab_result / bottleneck / report contracts
+- [references/report-template.md](references/report-template.md) —
+  the report skeleton with the mandatory sections
+- [references/subagent-contracts.md](references/subagent-contracts.md) —
+  prompts and invocation templates (acceptance criterion #2)

skills/npu-perf-optimization/references/feature-switch-matrix.md

@@ -0,0 +1,98 @@
+# Feature-Switch Matrix for Ascend NPU (A/B parameter space)
+
+Human-authoritative reference for the four switches evaluated by this skill.
+The machine-checkable subset lives in `run_ab_matrix.py::COMPAT_RULES`; when
+this document and the code disagree, fix one of them in the same PR.
+
+Primary source: `docs/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning.mdx`
+(official tuning guide incl. the 16×16 feature compatibility matrix) and the
+per-model best-practice pages.
+
+## 1. NPU Graph (task wording: "ACLGraph")
+
+Captures the compute graph via `torch.npu.NPUGraph` and replays it to remove
+kernel-launch overhead. Analogous to CUDA Graph. Best for decode with stable
+batch sizes.
+
+| Knob | Values | Notes |
+|---|---|---|
+| `--cuda-graph-bs <list>` | e.g. `8 12 24 36 48 …` | cover the batch sizes you actually serve; capture cost grows with the list |
+| `--disable-cuda-graph` | flag | the "off" variant; also required on dedicated prefill nodes |
+| `--cuda-graph-max-bs-decode N` | e.g. `4`–`64` | cap capture; align with observed serving batch distribution |
+
+A/B variants (P0): `off` vs `on` with a bs list derived from the workload's
+`max_concurrency` (e.g. include it and a few lower rungs).
+
+Constraints:
+- `--enable-torch-compile` and NPU Graph are **mutually exclusive** (set
+  `--disable-cuda-graph` when compiling).
+- `TASK_QUEUE_ENABLE=2` interferes with graph-launched tasks → use `0` for
+  decode-with-graph, `2` for throughput-critical prefill without graph.
+- Profiling with full Python-stack mapping needs `--disable-cuda-graph`;
+  graph-capture profiling needs `--enable-profile-cuda-graph` (traces go to
+  `SGLANG_TORCH_PROFILER_DIR/graph_capture_profile/`).
+
+## 2. DeepEP (MoE all-to-all)
+
+| Knob | Values | Notes |
+|---|---|---|
+| `--moe-a2a-backend` | `deepep` \| `ascend_fuseep` \| default | `deepep`: flexible prefill/decode paths; `ascend_fuseep`: general fused dispatch |
+| `--deepep-mode` | `normal` \| `low_latency` \| `auto` | `normal`: prefill/large token counts; `low_latency`: decode, graph-compatible; `auto` when unsure |
+| `SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK` | `0` (unbounded) … `1024` (hard cap) | must satisfy `max_running_requests * (1 + draft_tokens) <= value`; prefill: `0`, decode: match expected tokens/iter (e.g. `8`) |
+| `DEEP_NORMAL_MODE_USE_INT8_QUANT` | `1` \| unset | int8 dispatch quant: less comm volume, small accuracy trade-off; multi-node prefill mostly |
+
+A/B variants (P0, MoE models only): `baseline(no deepep)` vs `deepep+auto`;
+P1: `normal` vs `low_latency` under PD-mixed, and int8-quant on/off.
+
+Constraints:
+- EPLB (`--enable-eplb`) requires `deepep` (not `ascend_fuseep`).
+- DeepEP is MoE-only; do not add to dense-model variants.
+
+## 3. DP-Attention
+
+Attention layers run data-parallel while FFN/MoE stay TP/EP — removes KV-cache
+duplication and attention-side all-reduce during small-batch decode.
+
+| Knob | Values | Notes |
+|---|---|---|
+| `--enable-dp-attention` | flag | the switch itself |
+| `--dp-size` | 2, 4, … | must divide card count (remember A3: tp = 2 × cards); leftovers go to TP/EP |
+| `--enable-dp-lm-head` | flag | effectively required with DP-attention (avoids logits all-gather) |
+| `SGLANG_SCHEDULER_SKIP_ALL_GATHER=1` | env | safe under balanced load (`--load-balance-method round_robin`); drop if ranks look uneven |
+| `--moe-dense-tp-size 1` | with DP-attn | avoids pointless all-reduce on dense MLP when each DP shard has full weights |
+
+A/B variants (P0, tp ≥ 2): `off` vs `on (dp=2, lm-head, round_robin)`.
+
+Constraints: partial (🟠) compatibility with EP / chunked prefill / graphs —
+prefer changing one thing at a time; record the combination in the variant name.
+
+## 4. MTP / Speculative decoding
+
+| Knob | Values | Notes |
+|---|---|---|
+| `--speculative-algorithm` | `NEXTN` (DeepSeek-family MTP heads) \| `EAGLE3` (external draft; Qwen-family) \| `NGRAM` (training-free n-gram drafting) | NEXTN needs no draft weights; EAGLE3 requires `--speculative-draft-model-path`; NGRAM needs no draft model at all — use `--page-size 128` and chain verify only (`max_bfs_breadth 1`; tree verify needs page_size 1, rejected by the FIA kernel, and page 16 silently corrupts output), verified end-to-end on 910C, see https://github.com/sgl-project/sglang/pull/45797 |
+| `--speculative-num-steps` | prefill `1`; decode `2–4` | more steps → longer acceptance chains, more latency |
+| `--speculative-eagle-topk` | `1` (overlap scheduler) else `4–8` | `1` required with overlap scheduler (default on) |
+| `--speculative-num-draft-tokens` | prefill `2`; decode `4–5` | balance against latency budget |
+| `SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | env | overlap draft with verification (EAGLE v2/v3) |
+
+A/B variants (P0): `off` vs `on` with the model-family default (NEXTN steps=3
+draft=4 / EAGLE3 steps=3 topk=1 draft=4 — mirror the model's best-practice page).
+P1: steps ∈ {2,3,4} × draft ∈ {4,5}.
+
+Constraints: 🟠 with NPU Graph and chunked prefill; acceptance rate decides
+everything — if acceptance is low, MTP makes TPOT worse; report acceptance
+length when available.
+
+## Variant naming convention
+
+`<switch>_<key-knob>[_<p1-tag>]`, e.g. `npugraph_bs64`, `deepep_auto`,
+`dpattn_dp2`, `mtp_eagle3_s3d4`. The name must identify the single variable
+that differs from baseline (P0) or the interaction being explored (P1).
+
+## Sizing the matrix
+
+P0 (acceptance minimum): one on/off pair per applicable switch — 2 variants for
+dense models (Graph, MTP), 4 for MoE models on tp≥2. P1 only explores
+interactions where P0 showed |Δ| > noise floor. Every variant reuses the
+baseline workload verbatim.

skills/npu-perf-optimization/references/json-schemas.md

@@ -0,0 +1,117 @@
+# JSON Schemas (I/O contracts)
+
+All inter-phase handoff is JSON on disk. `schema_version` gates parsing; bump
+on breaking change. Unknown extra keys are allowed (forward-compatible) —
+readers must ignore what they don't know.
+
+## §A `server_config.json` (input to Phase 1 / matrix base)
+
+```jsonc
+{
+  "launch_command": ["python3", "-m", "sglang.launch_server", "…"],  // verbatim, caller-supplied
+  "env": {"HCCL_OP_EXPANSION_MODE": "AIV"},                          // merged over current env
+  "port": 6688,
+  "ready_pattern": "The server is fired up and ready to roll!",      // optional override
+  "startup_timeout_s": 900,
+  // meta (copied into artifacts; source of truth = the caller)
+  "hardware": "Atlas 800I A3", "cards": 1, "model": "Qwen/Qwen3-8B",
+  "quant": "W8A8(modelslim)", "tp_size": 2, "sglang_commit": "…"
+}
+```
+
+## §B `baseline/baseline.json` (Phase 1 output)
+
+```jsonc
+{
+  "schema_version": "1.0",
+  "meta": { /* hardware, cards, model, quant, tp_size, sglang_commit, date */ },
+  "config": { /* the server_config used */ },
+  "workload": { /* frozen bench spec, §F */ },
+  "metrics": {
+    "tpot_ms_p50": 37.2, "ttft_ms_p50": 120.5, "throughput_tok_s": 4123.4,
+    "completed": 256,
+    "runs": [ {"returncode":0, "log":"…", /* parsed per-run metrics */, "raw_json":"…"} ]
+  },
+  "sanity": {"short_gen_ok": true, "num_requests": 20}
+}
+```
+
+Invariant: `tpot_ms_p50` is the **median of ≥2 successful runs**; the raw runs
+stay embedded for audit.
+
+## §C `bottleneck.json` (Phase 2 output)
+
+```jsonc
+{
+  "schema_version": "1.0",
+  "parsed_dir": "…/*_ascend_pt/ASCEND_PROFILER_OUTPUT",
+  "total_duration_us": 0.0,
+  "category_share_pct": {"matmul": 41.2, "communication": 30.5, "…": 0},
+  "top_operators": [ {"name":"…","category":"matmul","count":312,
+                       "duration_us":0.0,"pct_total":12.3} ],
+  "stage_split_raw": {"<column>": 0.0},      // step_trace_time.csv columns, version-dependent
+  "attribution": "compute | comm | launch | mixed",
+  "hints": "switch guidance matching the attribution"
+}
+```
+
+## §D `ab/ab_result.json` (Phase 3 output)
+
+```jsonc
+{
+  "schema_version": "1.0",
+  "baseline_ref": "path/to/baseline.json",
+  "noise_floor_pct": 3.0,
+  "date": "…",
+  "variants": [
+    {
+      "variant": "npugraph_bs64",
+      "switches": { /* overrides applied */ },
+      "status": "ok | failed | rejected",
+      "metrics": {"tpot_ms_p50": 0, "throughput_tok_s": 0, "runs": [/* §B-style */]},
+      "vs_baseline": {"tpot_delta_pct": -8.4, "throughput_delta_pct": 9.1},
+      // failure-only:
+      "error": "…", "log_tail": "last 2KB of server log", "compat_errors": ["…"]
+    }
+  ]
+}
+```
+
+`state.json` (checkpoint) = `{"done": {<variant name>: <variant entry>}}`.
+
+## §E `reports/report.json` (Phase 4 output)
+
+```jsonc
+{
+  "schema_version": "1.0",
+  "date": "…",
+  "provenance": {"baseline": "…", "ab": "…", "bottleneck": "…", "meta": {}, "workload": {}},
+  "baseline_metrics": {"tpot_ms_p50": 0, "ttft_ms_p50": 0, "throughput_tok_s": 0},
+  "bottleneck": {"attribution": "…", "category_share_pct": {}, "top_operators": []},
+  "comparisons": [ {"variant":"…","status":"…","tpot_delta_pct":0,
+                     "throughput_delta_pct":0,"verdict":"human sentence"} ],
+  "recommendation": {
+    "overrides": {"launch_args": {}, "env": {}},
+    "compat_errors": [],                  // must be [] in a usable recommendation
+    "launch_command": ["…"], "env": {},
+    "expected": { "<variant>": {"tpot_delta_pct": 0} }
+  },
+  "noise_floor_pct": 3.0
+}
+```
+
+## §F frozen workload spec (embedded in §B/§D inputs)
+
+Keys mirror `sglang.bench_serving` args of the official best-practice pages:
+`dataset_name, backend, host, port, max_concurrency, num_prompts,
+random_input_len, random_output_len, random_range_ratio, seed`. Extra keys
+pass through as `--<dashed-key> <value>`. **Frozen rule**: once the baseline is
+collected, every variant must reuse the identical object (the matrix runner
+hard-fails on mismatch).
+
+## Parsing notes (bench output drift)
+
+`collect_baseline.py::METRIC_CANDIDATES` maps our canonical names
+(`tpot_ms`, `ttft_ms`, `throughput_tok_s`, `completed`, `duration_s`) onto
+version-drifting `bench_serving` JSON field names. Extend the candidate lists
+there when a new sglang version renames fields; never branch on versions.