Repository navigation
[rocprofiler-sdk] : collect SPM TA and VCA counters for gfx12 and gfx1250 - #13449
sauverma93 wants to merge 4 commits into
Conversation
✅ All Policy Checks Passed
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
9f51767 to
fb67bd9
Compare
fb67bd9 to
6a8dad6
Compare
44814c2 to
0daff31
Compare
178a2cd to
dff662a
Compare
bing-ma
left a comment
There was a problem hiding this comment.
Looks good to me; the gfx12/gfx1250 counter selection and the warm-I-cache SQC validation change both make sense.
TA_TA_BUSY is TA event 15 on gfx1250, matching gfx1200 and gfx1201. TA_TOTAL_WAVEFRONTS is TA event 32 on gfx1200, gfx1201, and gfx1250.
MI455 (gfx1250) reports TA activity as TX_VCA_VCA_BUSY and TX_VCA_TOTAL_WAVEFRONTS. The JSON tool requests those counters, and the validator selects them when the dispatch agent is gfx1250.
…unt. gfx1200 and gfx1201 count TA_TOTAL_WAVEFRONTS as TA event 32. Without that definition the SPM counter-value tests compare TA_TA_BUSY with a missing wavefront counter. The SQC hit and miss check divided by SQC_ICACHE_REQ. That ratio is valid on MI300, where the request count stays nonzero. On gfx12 and gfx1250 later dispatches report zero requests, hits, and misses once the instruction cache is warm, so the division fails. Compare hits and misses with the request count instead.
The rocprofiler-sdk formatting check rejects the previous line breaks.
64973f2 to
67178eb
Compare
PR Pundit review(Testing PR Pundit Code Review agent. Not all comments may be relevant. This is a one-time automated pass: please address what is useful, no need to request a re-review. Please reply to each point, whether it applies or not: we harvest these replies, and both kinds are enormously helpful for tuning the agent. Reach out to @peymanr for suggestions.)
1. 🟠 Should fix: gfx1250 VCA busy/wavefronts assert fails with bare AssertionError: this PR reuses message-less TA heuristic
Who is affected: Developers reading SPM test failures, mainly on gfx1250 where the VCA counters are now compared. Checked against the code by an independent pass. 2. 🟠 Should fix: SPM test passes with zero I-cache traffic on all dispatches: this PR replaced division check with
|
| assert float(busy) > float(wavefronts) | ||
|
|
||
| assert ( | ||
| 100 |
There was a problem hiding this comment.
Why do we remove this altogether, why not check if Req != 0 and compute the ratio in that case



Motivation
This follows up #12212 for gfx1250 and gfx12.
gfx1250 (MI455) reports TA activity as
TX_VCA_VCA_BUSYandTX_VCA_TOTAL_WAVEFRONTS. The JSON tool requestedTA_TA_BUSYandTA_TOTAL_WAVEFRONTS, so those TA counters never appeared in the gfx1250 samples.gfx1200 and gfx1201 do use the TA counters.
TA_TA_BUSYwas already defined.TA_TOTAL_WAVEFRONTSwas not, so the counter-value tests comparedTA_TA_BUSYwith a missing wavefront counter.The SQC I-cache check is the other failure, and it does not behave the same on every architecture. On MI300,
SQC_ICACHE_REQstays populated and hits and misses stay within that request count, so a miss-rate ratio is well defined. On gfx12 and gfx1250 these counters are nondeterministic per SQ and per bank. Early dispatches of a kernel still see I-cache traffic. Later dispatches of the same kernel can report no requests, hits, or misses, because the instruction cache is already warm. Dividing bySQC_ICACHE_REQthen fails.Technical Details
The JSON tool requests
TX_VCA_VCA_BUSYandTX_VCA_TOTAL_WAVEFRONTSalongsideTA_TA_BUSYandTA_TOTAL_WAVEFRONTS. It collects only the counters the agent supports.validate.pyreads the agent recorded on each dispatch:TX_VCA_VCA_BUSYandTX_VCA_TOTAL_WAVEFRONTSTA_TA_BUSYandTA_TOTAL_WAVEFRONTSThe VCA counters are already defined for gfx1250.
TA_TOTAL_WAVEFRONTSis now defined for gfx12, gfx1200, and gfx1201 on the same TA event used by gfx908 and gfx90a.The SQC check no longer divides by
SQC_ICACHE_REQ. It requiresSQC_ICACHE_HITSandSQC_ICACHE_MISSESto be no larger thanSQC_ICACHE_REQ. A dispatch with no I-cache traffic passes. A hit or miss reported with no requests still fails. A missing counter fails with its name.The tests this addresses:
test-spm-dispatch-collection-validate.test_spm_counter_valuestest-spm-dispatch-collection-validate.test_spm_counter_consistencytest-spm-dispatch-buffer-collection-validate.test_spm_counter_valuesIssue Tracking
ISSUE ID: AIPROFSDK-1151
Follow-up to #12212.
Test Plan
gfx1250
Re-run the SPM integration tests on gfx1250. The consistency checks pass.
Confirm the gfx1250 samples include
TX_VCA_VCA_BUSYandTX_VCA_TOTAL_WAVEFRONTS, and that the busy counter is greater than the wavefront counter.Re-run the two counter-value tests after the SQC comparison change:
test-spm-dispatch-collection-validate.test_spm_counter_valuestest-spm-dispatch-buffer-collection-validate.test_spm_counter_valuesThe VCA comparison already passed. The remaining failure was a warm I-cache: requests, hits, and misses were all absent, and the old check divided by the request count. That case is now accepted.
gfx12
TA_TOTAL_WAVEFRONTSis collected, andTA_TA_BUSYis greater than it.test-spm-dispatch-collection-validate.test_spm_counter_valuestest-spm-dispatch-collection-validate.test_spm_counter_consistencytest-spm-dispatch-buffer-collection-validate.test_spm_counter_valuestest-spm-dispatch-buffer-collection-validate.test_spm_counter_consistencySQC_ICACHE_*matches the expected gfx12 pattern. The first launches of a kernel still report requests, with hits and misses no larger than that request count. Repeat launches of the same kernel report no I-cache traffic. Both pass.rocprofv3-test-spm-json-validate.test_validate_spmstill requiresSQ_WAVESto match the PMC total exactly. SPM can come in slightly under that total, so this shader-clock check can still fail. The reference-clock run of the same check passed.Submission Checklist