Repository navigation
rocjitsu: use SIMD words and set-bit iteration for register sets - #12584
Merged
Merged
Conversation
❌ PR Check — Action Required
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
|
🚫 Please fix the failed policies before requesting reviews. The following policy checks failed:
The |
kuhar
reviewed
Oct 1, 2026
github-actions
Bot
added
the
documentation
bjacob
force-pushed
the
users/bjacob/register-set-word-iteration
branch
from
October 1, 2026 15:20
20669f7 to
8f3612b
Compare
kuhar
approved these changes
Oct 1, 2026
bjacob
force-pushed
the
users/bjacob/register-set-word-iteration
branch
from
October 2, 2026 19:04
8f3612b to
37ec9ee
Compare
bjacob
changed the title
Contributor
Author
|
@kuhar , thanks for another good comment: the B0-A0 translation workloads showed a minor regression , and pinning that down was very informative: in some cases, the old std::bitset code allowed the compiler to generate AVX2 instructions, and we had lost that. This highlighted that my PR was only halfway done. It needed to continue fully ditching std::bitset in favor of raw words, and then making the words be 256-bit on AVX2, and using a few explicit AVX2 intrinsics. Much more iteration ensued to make sure that the resulting code is neat while maximizing performance, and now we have this (Zen4 CPU): GCC 15
Clang 21
|
kuhar
reviewed
Oct 2, 2026
kuhar
approved these changes
Oct 3, 2026
RegisterSet iteration scanned every possible register position, even for sparse instruction def/use sets. Splitting std::bitset into 64-bit chunks improved iteration but also lost GCC -O2 AVX2 vectorization in bulk set operations used by B0-to-A0 translation. Store ordinary register classes in raw 256-bit SIMD words on AVX2 hosts, with a uint64_t fallback elsewhere. Implement masks, membership, counting, equality, and set algebra through word helpers without depending on standard-library representation or scalar byte order. Enumerate occupied SIMD lanes and set bits in ascending register order. Use consecutive-index batches for full lanes. Reload membership after callbacks and recover strictly beyond the emitted index when it changes, preserving forward additions and removals. Keep recovery on a cold path and unroll the full-lane loop for both GCC and Clang. Keep register class constant and inline available callback callees to improve alias analysis. Share range traversal between updates and queries, with partial-word prologue/epilogue and a full-word loop. Preserve clipping, single-bit fast paths, capacities, special-register separation, and the public API. Add and extend ten register-set regression tests covering bit masks, range boundaries, every byte pattern, partial class tails, set algebra, and callback edits within and across full lanes against independent bitset models. Keep benchmark artifacts outside the repository. Validation: full GCC release build; 542 focused tests and 107 analysis suite tests pass. The ten register-set tests also pass with GCC/Clang AVX2 and scalar builds, GCC AVX2 without BMI, and GCC ASan/UBSan. Five-trial standalone GCC/Clang benchmarks have 1,080 independently validated checksums. Across 36 equally weighted cases, the final iterator is 2.00x faster than the published PR on GCC and 1.63x on Clang (9.71x and 6.56x versus pre-PR). These are not whole-application startup estimates. Two runs per version over 20,130 CI corpus objects produced identical translation outputs, sizes, and diagnostics (120,780 translations). Mean B0-to-A0 translation time is 199.16s, versus 202.71s for the published PR and 205.06s before the PR, with matched GCC -O2 flags.
bjacob
force-pushed
the
users/bjacob/register-set-word-iteration
branch
from
October 5, 2026 13:46
5efecd1 to
6bc7120
Compare
bjacob
added a commit
that referenced
this pull request
Oct 5, 2026
Use the final PR #12584 implementation (6bc7120): header-only RegisterSetT with explicit StandardUint64 and Avx2M256 word selections, shared range operations, and mutation-aware set-bit iteration. Keep the public RegisterSet alias and all existing capacity and special-class rules. Compile and test both word implementations in AVX2 builds through the typed register-set suite. Use targeted callback inlining instead of recursive flattening, and mark WordArrayOps final as requested in review. Preserve the sanitizers branch's additional analysis tests and consumers.
bjacob
added a commit
that referenced
this pull request
Oct 5, 2026
Reconcile the refreshed PR #12584 register-set implementation and sanitizer resource accounting with develop's compute-queue and VM access changes. Regenerate ISA sources using the shared packed-VGPR resolver, retain the allocating realpath ABI and OSS7 support, and integrate legacy extended SDMA counts. Update tests for explicit register allocations and shader-array geometry, CDNA4 Wave64 execution, and independently backed scratch pages. Validation: full GCC build and CTest run (43,264 registered cases); fix all three failures and pass 471 affected tests on the rebuilt executable. Also pass 2,286 Python tests, GCC/Clang register-set typed tests, scalar UBSan tests, and repository-pinned pre-commit checks on the resolutions.
Contributor
TheRock Submodule Bump ActivityNewest first
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Register-set iteration scans all 106 SGPR, 1024 VGPR, and 1024 accumulator positions even for sparse instruction def/use sets. Visit occupied lanes and set bits while retaining SIMD words for bulk set operations.
Technical Details
Make the implementation header-only as
RegisterSetT<RegisterSetWordType>, selected by the enum valuesStandardUint64andAvx2M256, with aWordOpsspecialization for each.RegisterSetaliases the AVX2 specialization when available and the scalar specialization otherwise. Both implementations can be instantiated in an AVX2 build, so the scalar fallback is compiled and tested routinely.Use explicit word helpers for membership, masks, counting, equality, and set algebra. Share the partial-word prologue, full-word loop, and partial-word epilogue between range updates and queries. Preserve clipping, single-bit fast paths, register capacities, ordinary/special separation, and the existing
RegisterSetAPI.Iterate occupied lanes and set bits in ascending order, with consecutive-index batches for full lanes. Reload membership after callbacks to preserve forward additions and removals, keeping edit recovery on a cold path. Replace
gnu::flattenwith a small inline callback helper, a Clang call-site hint, and explicit inlining on the hot materialization/intersection callbacks. There is no dependency on private standard-library layouts or per-set dynamic allocation.Issue Tracking
None.
Test Plan
Exercise both word implementations in AVX2 builds and the scalar fallback without AVX2. Run the affected analysis and probe tests, sanitizers, and compiler checks. Compare the latest PR with the pre-PR baseline using independently checked microbenchmarks and matching corpus translations; keep builds and tests separate from timing.
Test Result
Performance
Without PR:
03d97e532a6. Latest PR:5efecd1b74c.Standalone iteration
Five randomized paired trials per case and implementation; 12 callbacks at three synthetic densities. GCC 15 and Clang 21,
-O3 -DNDEBUG -march=x86-64-v3, no LTO, Ryzen 9 7950X3D pinned to CPU 4. Each sample traverses 4,096 sets for 20 sweeps. All 720 baseline/latest-PR checksums match independently derived expectations.All cases have equal weight. These are callback-enumeration microbenchmarks, not whole-application or ConSan startup speedups. Each visited index remains observable through an opaque scalar compiler barrier. Projection callbacks also exercise updates; formatting includes allocation and string work.
The latest-PR benchmark applies the same explicit
always_inlineannotations as production to its materialization and intersection callbacks. Algorithms and inputs are unchanged; the without-PR version retains the original annotations.B0-to-A0 translation
Fresh measurements of the same 20,130-object CI package corpus. GCC 15.2,
-O2 -g -DNDEBUG -march=x86-64-v3, Ryzen 9 7950X3D, CPU 4. Two fresh processes per implementation, in baseline/PR/PR/baseline order. Builds and tests did not overlap timing, and the saved baseline binary was verified against its recorded SHA-256 hash.Timer scope: translator construction,
translate(), and destruction for already-loaded and parsed objects. File I/O, parsing, output hashing, and process startup are excluded. This is host translation time, not GPU execution or ConSan startup; no translation-result cache is used. The one CI-declared RCCL memory-limit case remains excluded.Values are means of summed per-object wall times. Two runs per implementation are not confidence intervals. Output byte counts, SHA-256 hashes, and diagnostic counts match for every input in both trials.
Submission Checklist