ci(atomesh): Add aggregated-service CI, AgentX runtime fixes and Kimi-K3 2P1D/agg cells

[wanzhenchn]

Summary

ATOMesh CI updates for the Kimi-K3 AgentX benchmarks:

• run an aggregated (non-P/D) server through the existing P/D pipeline
• add Kimi-K3 2P1D and aggregated cells
• capture torch profiler traces during the AgentX profiling phase
• report throughput and interactivity the way InferenceX does
• stop runtime kernel JIT stalls by reusing the FlyDSL and aiter artifacts the image already ships

Features

• Aggregated service in the P/D pipeline
  • A topology whose name starts with agg launches one server that answers both roles. The server reads the decode service block and env.decode, starts without a router, and gets no default KV connector.
  • pd_matrix.py derives a disaggregated flag from the topology name.
  • process_result.py and the dashboard count an aggregated cell's GPUs as that one server's, not prefill plus decode.
  • The metrics collector and report support a standalone server.
  • Adds kimi-k3-mxfp4-agg-tp8-dcp8-agentic-lmcache-1m-c72 as the aggregated counterpart of the 1P1D C72 cell.
• Kimi-K3 2P1D cells
  • Adds TP8/DCP8 agentic cells at C72, C128 and C256.
  • When more than one prefill instance runs, the router pins each session to one prefill (--prefill-policy dp_sticky, keyed on X-Session-ID), so later turns land on the prefill that holds their prefix.
• Torch profiler traces in the AgentX profiling phase
  • Adds a dispatch-only profile suite with kimi-k3-mxfp4-1p1d-tp8-dcp8-agentic-lmcache-1m-c72-torch-profile.
  • ATOMESH_TORCH_PROFILE_START, ATOMESH_TORCH_PROFILE_DURATION and ATOMESH_TORCH_PROFILE_ROLES open a start_profile/stop_profile window on the prefill and decode servers.
  • Per-rank traces are uploaded as artifacts and linked in the job summary.
• atom_source dispatch input
  • mounted (the default) runs the checkout; image runs the copy installed in the image (/app/ATOM).
  • The launcher checks which atom package the servers will import and fails fast if it does not match the choice.

Fixes

• Slurm picks idle nodes on atomesh-cicd. Removes the hard-coded ATOMESH_NODE_POOL and its membership checks. auto lets Slurm choose idle nodes; an explicit node list still pins the job.
• Kimi-K3 AgentX cells aligned with the recipe (fix(ci-mesh): K3 AgentX cells — FP8 prefill attention and capture from bs=1 #2417).
  • FlyDSL FP8 prefill attention (ATOM_USE_FLYDSL_FP8_PREFILL_ATTN=1) on every band.
  • cudagraph: auto captures from bs=1 instead of bs=2.
• Reuse the image's AOT FlyDSL kernels.
  • The image ships every tuned FlyDSL kernel AOT-compiled, but the files are root-only, so the --user container could not read them. Each new prefill M bucket then compiled at runtime and stalled the engine for 8-60 s.
  • The kernels are now extracted once per node, uid and image (~90 MB) into a shared cache, and workers point FLYDSL_RUNTIME_CACHE_DIR at it.
  • If the shared cache is unavailable, workers copy the kernels into a private directory or, as a last resort, JIT-compile them.
• Link the image's prebuilt aiter modules into each worker's AITER_JIT_DIR.
  • With AITER_JIT_DIR set, aiter imports only from that directory, so an empty one made every worker rebuild modules the image already ships (for example, a 300 s stall at the first CUDA-graph capture).
  • The image's .so files are now symlinked into the writable directory. Modules the image lacks still build there.

Benchmark reporting

• New "P90 Interactivity" column, computed as 1/p90(ITL). It shows -- for fixed ISL/OSL rows.
• Renamed agentic column. It is now "P90 E2E Normalized Interactivity", computed as 1/p90(ITL + TTFT/OSL). The dashboard reads both the old and the new header.
• Agentic throughput computed as in InferenceX's compute_throughput_stats. Input, output and total tok/s are the summed ISL/OSL of the successful profiling records, divided by the span from the first request start to the last request end. AIPerf's time-sliced throughput reads lower on agentic runs.

Kimi-K3 case changes

• Enables PrefillDelayer on the aggregated C72 cell (ATOM_PREFILL_DECODE_INTERVAL=4, ATOM_PREFILL_DELAYER_MAX_QUEUE_MS=5000) to match InferenceX.
• Adds a 1P1D C88 cell: max_num_seqs: 176 and the 192 GiB CPU tier used from C56 up.
• Drops the 1P1D C40 and 2P1D C64 cells. The 2P1D anchor moves to C72, and C72/C128/C256 resolve unchanged.
• Syncs the case_names help text with the suites: removes C12, C14 and C80, which no suite defines, and adds the aggregated C72 cell.

Testing

• pytest tests/test_atomesh_*.py: 57 passed, 70 subtests passed.
• models_atomesh.yaml and atomesh-benchmark.yaml parse at every commit, and the Kimi-K3 case_names list matches the defined cells one-to-one.
• pd_matrix.py --suite nightly --case kimi-k3-mxfp4-1p1d-tp8-dcp8-agentic-lmcache-1m-c88 resolves to one cell.
• CI runs: https://github.com/ROCm/ATOM/actions/runs/38052766052

[github-actions]

🏷️ CI Guide

Runs automatically on every eligible PR before approval:

• ✅ Pre Checkin: Black, Ruff, catalog schema validation, non-GPU unit tests

Heavy model tests:

• ✅ Run after the PR is approved and Pre Checkin passes
• ✅ Run immediately when an approval review is submitted
• ✅ Can be requested before approval with labels

Label	Tests
ci:full	Run all heavy PR model tests: native ATOM, vLLM, and SGLang
ci:atom	Run native ATOM model accuracy tests
ci:vllm	Run ATOM vLLM OOT model accuracy tests
ci:sglang	Run ATOM SGLang model accuracy tests

Heavy jobs are skipped when the PR is not approved and no matching ci:* label is present.
Add labels via the sidebar or gh pr edit 2528 --add-label <label>