Repository navigation
recipe(kimi-k3): add MI455 B0 wide-EP optimization workflow - #2446
Closed
XiaobingSuper wants to merge 20 commits into
Closed
XiaobingSuper wants to merge 20 commits into
XiaobingSuper wants to merge 20 commits into
Conversation
First gfx1250 recipe in the tree, and the first to document the B0->A0 code-object translation these parts require. Full-size Kimi-K3 across four gfx1250 nodes, 4 GPUs each: -tp 1 with dp16/ep16 and DP attention, Triton MLA, MXFP4 routed experts. Validated at GSM8K strict-match 0.9545 (lm_eval, 5-shot, full 1319). The translation section leads the page because it is the only part with no safe default. These are B0 parts running kernels compiled for A0; if the rocjitsu hotswap prefix is not loaded, every generated token is `!` and GSM8K is 0% -- with HTTP 200, finish_reason "stop", and healthy throughput and TTFT throughout. The image's own rocjitsu is not a substitute: its hook installs and logs normally but its translation store is empty, so it translates nothing. The page gives the sizes and store count that distinguish the two, the runtime log lines that prove translation happened, and states that HOTSWAP=0 is not a fallback. Also recorded: the fabric-domain precondition (mismatched PPOD/VPOD hangs rather than errors), the KV-budget arithmetic that makes FULL_DECODE_ONLY necessary rather than preferable, why max_tokens must be generous for a reasoning model whose content field empties out when the thought is truncated, and an accuracy gate to run before any benchmark -- both of the platform's headline failures are silent, and one of them benchmarks well. Throughput is explicitly listed as not characterized rather than estimated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com>
… cause
Rename for consistency with the other Kimi-K3 recipes in the directory.
Correct the cudagraph claim. The page said FULL_DECODE_ONLY plus
--max-num-batched-tokens 2048 were both needed to make available_for_kv
positive. Only the token budget matters on the native engine:
- capture_cudagraph() bounds capture with
max_schedulable_decode_bs(...), so ATOM's manual FULL capture is
already decode-only and prefill never replays a graph;
- nothing in the native runtime calls mixed_mode(), which with
separate_routine() is the only predicate where FULL and
FULL_DECODE_ONLY differ -- every other one (decode_mode,
requires_piecewise_compilation, max_cudagraph_mode,
has_full_cudagraphs, is_attn_ffn_piecewise) agrees;
- _estimate_cudagraph_overhead() takes the same non-piecewise branch
for both and derives the pool from the allocator high-water mark set
by the profile run, which --max-num-batched-tokens sizes.
So the recipe now uses the default FULL and attributes the fix to the
token budget alone, with a note that the validated run happened to pass
FULL_DECODE_ONLY and that the mode IS honored under the vLLM plugin
backend, where vLLM's runner reads mixed_mode().
This also drops one of the two unmerged changes the recipe depended on.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The recipe described ATOM_UNFUSED_GATHER_KV_B_PROJ as a knob to set. On stock ATOM that env var does not exist, so following the page as written gets a server that starts, serves short prompts, and then dies on the first long prompt at concurrency -- the LLVM PHI assertion the patch exists to avoid. Say so where a reader will hit it: a row in the validated-configuration table, the prerequisites list, a dedicated section with the fetch command and the failure it prevents, an inline note on the env var itself, and the known-issues row. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The page checked PPOD_ID/VPOD_ID and then went straight to bringing up four nodes. That check only says the nodes are in the same vPOD, not that the fabric between them works -- and the expert all-to-all rides that fabric, where a bad link hangs rather than errors. At the server level that is indistinguishable from the intermittent ncclCommInitRank hang already in the known-issues table, so without a fabric test the first response to a stuck rendezvous is to retry a 25-minute cold start. Adds 07_ualoe: what it tests and why 06_interconnect_bandwidth cannot (it crosses an OS-image boundary via HIP fabric VMM handles), build, the correctness run, the bandwidth run, and the measured numbers (read 2029 / write 3192 / bidir 3986 GB/s aggregated over 4 GPU pairs) that put cross-node EP in the same order as intra-node XGMI. Records that ~4.5 GB/s instead means MNNVL is off and traffic fell back to TCP, and that this firmware reports ACCEL_STATE ACTIVE where the suite's README says READY. Also the three failure modes that cost real time: aggregate mode takes every GPU on both nodes; a dead peer leaves the listener holding memory so the next round fails silently; and fabric errors appear only in dmesg, with the AccId mapping to find the offending node and GPU. The cleanup command is spelled out because the obvious `pkill -f ualoe` matches the user's own command line and kills their shell. Cross-links the two known-issues rows that this test discriminates between, and notes that amd-smi fabric's zeroed BANDWIDTH/LATENCY fields are unpopulated rather than measured. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was one bullet saying amdgpu is blacklisted and that accel_state goes active eventually. It is a prerequisite for the fabric test that follows it -- an untrained link makes ubench07 fail exactly like a fabric fault -- so it now says what to load, what to check, and how to tell the two apart. Records the recommended modprobe parameters and why they matter for bring-up: with the defaults (gpu_recovery=-1, halt_if_hws_hang=0) a hardware-scheduler hang triggers a recovery reset, so a failed run leaves nothing in the log to explain itself. On a 25-minute cold start with this many silent failure modes, that is expensive. They are load-time parameters, so the page also shows how to read back what is actually loaded. Adds local_accels and station_lane_en_bitmap alongside accel_state. accel_state is the gate -- every entry must read active. The other two say how the link came up rather than merely that it did, so a partially trained station shows as a bitmap that differs from its peers. Their values vary with partitioning, so the page says to record the healthy values per node and diff after a reboot rather than matching a fixed string. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The page said to fetch the PR branch and check it out. That is the wrong instruction for this platform: the PR is based on main, while the gfx1250 image carries its own bring-up build of ATOM that is not main. Checking out the branch replaces that build and silently drops whatever bring-up deltas the image was made with, leaving you debugging a different engine than the one the recipe was validated against. Replace it with: take the diff, keep the image's ATOM. Fetch pull/2380.diff, filter it to the three runtime files (the other three are tests the image does not ship), resolve the installed package via atom.__file__ so it works for an editable checkout or a site-packages wheel, dry-run, then apply. Adds a verification step, because the failure is half silent: if envs.py alone fails to patch, ATOM_UNFUSED_GATHER_KV_B_PROJ is ignored at runtime with no complaint, which looks exactly like not having patched at all. Also notes this is container filesystem state, so docker rm discards it -- the same footgun the rocjitsu prefix has. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…setup Throughput: first fixed-length numbers for this configuration (1k/1k and 14k/500 at concurrency 64), with the reading that matters -- TPOT at concurrency 64 equals single-stream TPOT, so --max-num-seqs 8 is leaving throughput on the table. Also documents that benchmark_serving silently reports all-zero metrics when --served-model-name is omitted: the client sends the filesystem path as the wire name, every request 404s, and the run 'succeeds' in under a second. Profiling: ATOM_TORCH_PROFILER_DIR / --torch-profiler-dir on the server plus --profile on the client. The client flag alone is a silent no-op. AgentX: pinned aiperf version and commit, dataset, full command, and the two failures that happen before the first request -- --tokenizer must be the HF repo id, and K3's tokenizer.chat_template is None by design because tokenization_kimi.py implements apply_chat_template() as a method. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ot work The previous revision claimed ATOM_TORCH_PROFILER_DIR and --torch-profiler-dir were equivalent. They are not, on this image: with the env var the engine reports 'torch_profiler_dir': None and writes nothing, while /start_profile still returns HTTP 200 and the benchmark still prints a full metrics table. The only symptom is an empty directory. atom/plugin/config.py hardcodes torch_profiler_dir=None in three places, which is the likely cause. Adds the one-line log check that catches it before a run is wasted, the verified per-rank output layout and sizes, and the top GPU kernels from a 14k-prefill run -- expert all-to-all (dispatch + combine, 14.9M us) costs more than the largest GEMM (11.2M us), and that GEMM names itself bandwidth_bound. Also adds the 14k/32 column and states explicitly that benchmark_serving's throughput figures are cluster aggregates, not per-GPU. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ook right and are not Records the server config AgentX actually needs: prefix caching on and --gpu-memory-utilization 0.94 (available_for_kv 33.53 -> 50.97 GB, +52%), with --max-num-batched-tokens and --max-num-seqs left alone. Documents why raising --max-num-batched-tokens is the wrong instinct: it does not boot (KV budget goes to -4.67 GB at 16384/0.93, and no reachable utilization covers a ~40 GB deficit), and chunk size is not the reason agentic prefill is slow -- at the same 2048 chunk a 14k fixed-length run does 1252 tok/s per rank against 283 tok/s on the trace, because attention cost is quadratic in prompt length. Also records the prefix-caching deadlock seen at 0.90, with the detail that makes it diagnosable: 15 of 16 ranks logged the _sync_dp_state timeout and the one that did not was the rank at 98.6% KV occupancy -- a collective missing a participant, not a slow cluster. The client showed errors=0 throughout. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…triage Measured on a 4-node gfx1250 rack (16 GPUs, 432 GiB boards) with rocm/fw-bringup:gfx1250-atom-20260918-ep8. - B0→A0 translation is image-dependent, not silicon-dependent. That image's kernels are already built for B0: the hook installs on all 16 ranks and performs zero translations, and GSM8K still scores 0.9621 +/-0.0053 over the full 1319. Adds an A0-image vs B0-image table and a 60-second test to tell the two apart, since no revision field on the board reports the stepping. - HBM per GPU is board-dependent (288 and 432 GiB both exist); every KV number now says to re-derive it. available_for_kv measured at 159 GB @0.90. - accel_state can stick at `unconfigured` permanently and look like "still training"; records the recovery and the ifoe ordering hypothesis. - Verify the patch and prefix by byte size: a 0-byte .py from an interrupted docker cp imports cleanly and passes every check in the current text. - PR #2380's envs.py hunk does not apply to this image's ATOM; says where to put the block by hand. - Adds the fabric-down signatures seen while accel_state read `active` (HELLO/IMPORT/XA, p2p_tmp.cc:372) and notes the exporter-side dmesg is empty. - Adds a minimal torch.distributed all_reduce to separate a platform fault from the intermittent ncclCommInitRank hang, plus py-spy is not in the image. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The "do NOT raise it" advice is correct for 288 GiB boards and wrong for 432 GiB ones. Measured at 16384 / --gpu-memory-utilization 0.94: available_for_kv 118.80 GB (still 3.5x a 288 GiB board at 0.90), peak_torch 194.67 -> 236.08 GB, cudagraph_est 1.18 -> 9.40 GB, and TTFT on a short prompt 2.73 s -> 0.50 s. It is not an optimisation there, it is what makes AgentX measurable at all: at 2048 the warmup needed 2h52m for 707 requests and the 1800 s profiling window retired 3 of 89 trajectories, which aiperf rejects as ProfileMetricCoverageError (95% coverage required) -- documented with its signature, including that the "Benchmark Duration: 3.63 sec" it prints is the span between the completed requests, not the run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
First AgentX result on this configuration, filling the section that said "not yet reported". 1790 s window, 53 trajectories retired, 0 errors, coverage passed (ITL 99.4%). Headline: interactivity p90 2.53 tok/s/user, 182 tok/s/GPU total throughput (2913 aggregate over 16 GPUs), 92.36% measured prefix-cache read rate against a 94.68% theoretical hit rate for the trace. Full TTFT/ITL/ISL/OSL percentiles included. Three caveats stated with the numbers: the run used --fake-eplb so the model output is not valid and the figures are not a clean upper bound either (uniform routing gives the receiving rank smaller per-expert batches than real skewed routing); the launch used --max-num-batched-tokens 16384 and --concurrency 32 rather than the values in the AgentX launch section; and there is no baseline, so nothing can be attributed to any single knob. Decode is the bottleneck, not prefill: 212 output tokens cost ~326 s at 1.4 s per token while prefill aggregates 2907 tok/s. Effective concurrency settles at 15 against a nominal 32. Also documents ATOM_DP_SESSION_AFFINITY=1, which pins a session to its cache owner so a later turn does not re-prefill on another rank. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… sizing - Reframes --fake-eplb. Agentic runs on this configuration are measured with it so expert load is uniform and successive runs compare; the generated text is not meant to be read, and correctness belongs to the accuracy gate and GSM8K. Keeps the technical note that uniform routing is not simply a best case -- it gives the receiving rank smaller per-expert batches than real skewed routing. - Records that effective concurrency is 15.11 against a nominal 32, so --concurrency 16 and 32 land on nearly the same working point and sweeping downward buys nothing; to move the operating point, go up. - Notes EP32 as the obvious next lever: per-GPU expert weight halves (0.92 -> 0.46 GiB/layer), freeing ~41 GiB per GPU for KV. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A job whose containers exist but have not yet loaded weights reads 0% on every GPU, so a pre-flight that only reads rocm-smi will call the rack idle and start on top of someone else. It surfaces minutes later as a negative KV budget -- which is the engine refusing correctly, not a configuration problem. Adds the signature and points the check at the full container list and docker events, which show a create that no GPU counter reflects yet. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous commit (46bee30) carried this message but was empty -- a script error pushed unchanged content. This is the actual change. A job whose containers exist but have not yet loaded weights reads 0% on every GPU, so a pre-flight that only reads rocm-smi calls the rack idle and you start on top of someone else. It surfaces minutes later as a negative KV budget, which is the engine refusing correctly rather than something to fix by raising --gpu-memory-utilization. Adds an "Is the rack actually free?" section with the signature and a check built on the full container list and docker events -- a create shows up there before any GPU counter moves. Notes that filtering docker ps by your own container name answers a different question than the one being asked. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The working config was spread across three sections as deltas. This adds one self-contained appendix so it can be copied rather than reassembled. Records the rack (B0 silicon, 432 GiB boards, 16 GPUs, image gfx1250-atom-20260918-ep8, no B0->A0 translation needed), the per-node container and patch setup with the byte sizes to verify, and two configs with the results each produced: - Config A, stock launch (2048 / 0.90 / no prefix cache): GSM8K 0.9621 strict-match over the full 1319 in 9 min, available_for_kv 159.37 GB, cold start under 6 min. - Config B, agentic (16384 / 0.94 / prefix caching / --fake-eplb / ATOM_DP_SESSION_AFFINITY=1): full env block and server command verbatim, the aiperf invocation, available_for_kv 118.80 GB, TTFT 0.50 s. Notes that the runs still had HSA_TOOLS_LIB set and it did nothing (zero translations across 16 ranks), so those lines can be dropped on a B0-built image; that --warmup-requests-per-lane moves warmup between 2h52m and 13.5 min and must be held constant across runs being compared; and that the InferenceX repo moved aiperf to inferencex-e2e/utils/aiperf with no requirements.txt. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…umbers The page was written from the A0-image point of view, where the story is translation and the goal is getting the model to run at all. On rocm/fw-bringup:gfx1250-atom-20260918-ep8 the kernels are built for B0, the translation layer is dead weight, and what matters is performance. Reorders the page accordingly. - New opening section names the image explicitly and contrasts A0-built (translation mandatory, functional bring-up) against B0-built (no translation, the configuration worth measuring), then leads with the numbers: agentic interactivity p90 2.53 tok/s/user, 182 tok/s/GPU total, 92.36% prefix-cache read, GSM8K 0.9621, cold start under 6 min. - Marks the existing fixed-length throughput table as A0-path, not measured on this image. - New "Where the time goes, and what to try next": decode is the bottleneck by a wide margin -- 212 output tokens cost ~326 s against ~35 s of prefill for 100,926 input tokens, ITL p50 1,898 ms, and effective concurrency of 15.11 across 16 ranks means every decode step pays the full MoE all-to-all for a batch of about one. Ranks the levers: --max-num-seqs (never swept, 118.80 GB of KV available), concurrency upward, EP32 (frees ~41 GiB/GPU), and real --eplb-enable as distinct from --fake-eplb. Notes prefix caching is already at 92.36% of a 94.68% ceiling and is not the thing to tune next. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…nd B0 Corrects an error introduced earlier in this branch. There are not two images: A0 and B0 machines run the SAME image, rocm/fw-bringup:gfx1250-atom-20260918-ep8, and both need ATOM PR #2380. The image ships B0 code objects -- which is why the library is librocjitsu_gfx1250_b0_to_a0.so -- so translation is what A0 silicon needs to run them, and B0 needs nothing. The previous text described this as "A0-built vs B0-built images", which was wrong. Restructures the page around the B0 path, which is the one with numbers: - New opening states the one difference, points at the library name as the reason, and adds a "where to go" map -- the page had no way to tell where the B0 material started. - The translation chapter moves out of position two to Appendix B, "running the same image on A0 silicon", opening with "only read this if your parts are A0". Keeps the wrong-prefix table, install steps and verification intact, and keeps the behavioural test for telling the two apart, since no revision field on the board reports the stepping. - The validated configuration becomes Appendix A and is linked from the top. - Headline numbers now include the fixed-length 14k/16 con32 point: 26,207 tok/s total, TTFT median 13.2 s, TPOT median 73.9 ms. All 18 internal links verified against GitHub's anchor rules. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Documents the state of the two MoE dispatch backends, from source, without a measurement. MEGA_DISPATCH takes flydsl or mori and aiter's own default is flydsl, so the recipe naming mori looks like a tuning choice. It is not, and on the fp4 wire the variable does not decide anything: ATOM passes dispatch_backend="mori" explicitly whenever MEGA_DISPATCH_WIRE is fp8 or fp4, and an explicit argument wins over the env read. Setting MEGA_DISPATCH=flydsl changes nothing. The reason is the scale row. fp4 and fp8 carry a per-token scale next to the payload and only mori's dispatch kernel moves that row, which both ATOM and aiter state in comments. Using flydsl therefore means dropping to MEGA_DISPATCH_WIRE=bf16, a 4x wider dispatch payload -- against a stack where expert all-to-all is already the second-largest cost (10.08M us of mori_ep_dispatch_tdm_fp4x2 vs 11.16M us for the largest GEMM). Recorded as not-measured, so nobody spends a run finding out the env var alone is inert. Points at ATOM_MORI_V2_FUSED as the knob that is live on the fp4 wire and has not been swept. Also flags MEGA_WIRE as deprecated: renamed to MEGA_DISPATCH_WIRE, no longer read, and both ATOM and aiter raise if the two disagree. Existing scripts survive only because the values match. Removed from the launch blocks. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
🏷️ CI GuideRuns automatically on every eligible PR before approval:
Heavy model tests:
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| @@ -0,0 +1,91 @@ | |||
| #!/usr/bin/env python3 | |||
Contributor
Comment on lines
+4
to
+11
| from __future__ import annotations | ||
|
|
||
| import argparse | ||
| import csv | ||
| from pathlib import Path | ||
|
|
||
|
|
||
| M_VALUES = (1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, 8192, 16384) |
Contributor
There was a problem hiding this comment.
Import block is un-sorted or un-formatted
Suggested change
| from __future__ import annotations | |
| import argparse | |
| import csv | |
| from pathlib import Path | |
| M_VALUES = (1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, 8192, 16384) | |
| from __future__ import annotations | |
| import argparse | |
| import csv | |
| from pathlib import Path | |
| M_VALUES = (1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, 8192, 16384) |
Contributor
Author
|
Closing per scope: this is a recipe/reproduction record and experimental artifacts, not a production optimization change. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Validated result
With the workload held fixed at 1,024 input / 256 output tokens and concurrency 128, enabling
ATOM_MORI_V2_FUSED=1produced:The fixed AgentX target is B0 K3 DP16/EP16 con32. Its 900-second baseline is 2.9708 tok/s/user p90 interactivity; the optimization target is >=4.456.
Validation
git diff --checkDraft status
This stays draft because the final four-node AgentX A/B is blocked by node availability and recurring gfx1250 no-retry page faults. Isolated BF16, ptpc_fp8, checkpoint, and quantized-combine results are documented but are not presented as AgentX E2E gains.
The gfx1250 long-prefill configuration requires #2380. Quantized combine additionally requires ROCm/AITER #5176 or newer.
Made with Cursor