feat(mesh-ci): report p90 latency and AIPerf raw metrics in benchmark summary and support kimi-k3 agentic ci

[wanzhenchn]

CI

Benchmark matrix — .github/benchmark/models_atomesh.yaml

• All 11 concurrency points from recipes/Agentic-Kimi-K3.md at (recipe) add Kimi-K3 AgentX recipe #2131 (30c2f1b): 1/2/4/8/12/14/16/32/40/56/64 across the three serving bands.
• Verified field by field against the recipe's per-concurrency table: DCP, spec tokens, acceptance length, LMCache size, ReplaySSM, max-num-seqs, batched tokens, GPU utilization, graph_max.
• CUDA graphs: --cudagraph-mode FULL --level 3 over the dense range [2 … 2*CONC*(1+spec)].
• ReplaySSM off on C1/C2/C4, on for C8–C16, off from C32 up. AITER_REUSE_IDENTICAL_COMM_GROUPS only on C56/C64.
• LMCache 128 GiB per rank on C8–C40, 192 GiB on C56/C64, with explicit ATOM_NUMA_NODE=0,0,0,0,1,1,1,1 and auto-binding off.
• Throughput band drops the CPU state-offload tier: it keeps no draft model and no ReplaySSM, so OFFLOAD_STATE_CPU_SIZE was taking 32 GiB per rank from the paged KV for a state pool nothing replays from.
• --state-checkpoint-interval-tokens -1 and ATOM_STATE_CHECKPOINT_DEMAND=0 both differ from the ATOM defaults; omitting either reproduces a different configuration than the recipe's published numbers.

Launcher plumbing — .github/scripts/atomesh/pd_server_atom.sh, pd_submit.sh, pd_slurm_job.sh

• CUDA-graph capture sizes derived from concurrency and spec width instead of enumerated per case.
• cudagraph_max_num_seqs available as a per-case override — C14 pins the window to 32, since 2*CONC gives 28 and the server dies during graph warmup.
• DCP, speculative-decoding and state-checkpoint parameters propagated per role; STATE_CHECKPOINT_ allowed through the env forwarding list that gates what reaches the container.
• server_args self-validating: the export mapping records which keys it reads and the submit step fails on any key it never consumed, so a config key can no longer be dropped without a trace.
• GSM8K joins SWE-bench in getting its own service phase, so the accuracy server restarts with its own arguments rather than reusing the benchmark server.

Result reporting — .github/scripts/atomesh/process_result.py, aiperf_console_report.py, .github/workflows/atomesh-benchmark.yaml, .github/dashboard/index.html

• Mean / p90 / p99 for TTFT, TPOT and E2E plus TP / DCP / Spec, replacing single averaged numbers that hid the tail dominating agentic traces; p90 exported at the source and carried on every dashboard point.
• New aiperf_console_report.py reproduces AIPerf's console tables — prefill/decode throughput, tokens in flight, CO-aware latency — otherwise untracked; no-op for fixed ISL/OSL runs.
• Each case's log path surfaced in the job summary; performance table and log-path table both ordered by (benchmark_model_name, max_concurrency).
• Dashboard details popover builds parallelism and percentile fields on the markdown-summary row path, and no longer reports a hardcoded expert-parallelism value it never measured.

[github-actions]

🏷️ CI Guide

Runs automatically on every eligible PR before approval:

• ✅ Pre Checkin: Black, Ruff, catalog schema validation, non-GPU unit tests

Heavy model tests:

• ✅ Run after the PR is approved and Pre Checkin passes
• ✅ Run immediately when an approval review is submitted
• ✅ Can be requested before approval with labels

Label	Tests
ci:full	Run all heavy PR model tests: native ATOM, vLLM, and SGLang
ci:atom	Run native ATOM model accuracy tests
ci:vllm	Run ATOM vLLM OOT model accuracy tests
ci:sglang	Run ATOM SGLang model accuracy tests

Heavy jobs are skipped when the PR is not approved and no matching ci:* label is present.
Add labels via the sidebar or gh pr edit 2177 --add-label <label>