Skip to content
Merged
Show file tree
Hide file tree
Changes from 248 commits
Commits
Show all changes
251 commits
Select commit Hold shift + click to select a range
1bc8bee
Add SBRR partial pass kernel generator.
eng-flavio-teixeira Feb 13, 2025
2b0c5d3
- Remove fixed length-64 partial pass kernels.
eng-flavio-teixeira Feb 14, 2025
3356f9f
- Remove no longer needed code.
eng-flavio-teixeira Feb 14, 2025
4e56a6d
Fix register size in partial pass sbrr
eng-flavio-teixeira Feb 15, 2025
bc281f9
- Fix offset initialization.
eng-flavio-teixeira Feb 18, 2025
ed2afe3
- remove debug code.
eng-flavio-teixeira Feb 18, 2025
97e6379
- Clean up debug code.
eng-flavio-teixeira Feb 18, 2025
e7dceae
- Handle direction in partial-pass twiddle multiply step.
eng-flavio-teixeira Feb 18, 2025
bd3af1b
- Fix partial pass radix include in rtc_stockham_gen.
eng-flavio-teixeira Feb 20, 2025
9c021bf
- Clean up comment
eng-flavio-teixeira Feb 20, 2025
860cc78
- Disable half-lds support in partial-pass SBRR kernels.
eng-flavio-teixeira Feb 21, 2025
ec4eb38
- Fix partial-pass kernel name
eng-flavio-teixeira Feb 22, 2025
d50ae30
Fix formatting.
eng-flavio-teixeira Feb 24, 2025
c6c02d0
- Add partial pass test scripts.
eng-flavio-teixeira Mar 7, 2025
f6a26a0
- Fixes to test scripts
eng-flavio-teixeira Mar 8, 2025
64710e7
WIP
eng-flavio-teixeira Apr 10, 2025
6bd5960
- Merge with develop
eng-flavio-teixeira Apr 23, 2025
de36c9f
- Resolve merge conflict.
eng-flavio-teixeira Apr 23, 2025
b368f0b
- Clean up.
eng-flavio-teixeira Apr 23, 2025
e89612c
- Delete partial-pass test scripts.
eng-flavio-teixeira Apr 23, 2025
6f286b0
- undo test changes.
eng-flavio-teixeira Apr 23, 2025
987a7f9
- Clang formatting.
eng-flavio-teixeira Apr 23, 2025
7b092a1
- Further clean up
eng-flavio-teixeira Apr 24, 2025
5c11172
- Remove no longer needed include
eng-flavio-teixeira Apr 24, 2025
708a98c
- More clean-up.
eng-flavio-teixeira Apr 24, 2025
47cbc4e
- match SBRR configuration.
eng-flavio-teixeira Apr 25, 2025
eecba0a
- Changes to kernel-generator.py and function pool to support partia…
eng-flavio-teixeira Apr 25, 2025
acfe158
WIP
eng-flavio-teixeira May 8, 2025
ce4e4d4
formatting.
eng-flavio-teixeira May 8, 2025
4ae3e02
- Fix ROCFFT_LAYER=8 not displaying the new partial-pass nodes.
eng-flavio-teixeira May 8, 2025
96ee9cf
- WIP: Fixes for dealing with column- to row-major configuration ent…
eng-flavio-teixeira May 15, 2025
64b6cd4
- Merge with develop
eng-flavio-teixeira May 16, 2025
eedc32d
- Resolve merge conflicts.
eng-flavio-teixeira May 16, 2025
834f888
- Formatting.
eng-flavio-teixeira May 16, 2025
66b16bb
- Refactor kernel-generator.py changes.
eng-flavio-teixeira May 16, 2025
e2c142b
- Refactor and improvements.
eng-flavio-teixeira May 16, 2025
304dd85
- Refactor function pool.
eng-flavio-teixeira May 16, 2025
0a53c1e
- Refactoring.
eng-flavio-teixeira May 20, 2025
a37b90c
- Get partial pass off-dim from function pool.
eng-flavio-teixeira May 21, 2025
bbbe036
- Clean up.
eng-flavio-teixeira May 21, 2025
6c0dfb2
- Further partial-pass kernel config validation.
eng-flavio-teixeira May 21, 2025
8bdad2f
Merge with develop
eng-flavio-teixeira May 27, 2025
aae9b33
- Add further validation for kernel-generator.py partial-pass data.
eng-flavio-teixeira May 27, 2025
20f1d3c
- Remove no longer needed field from KernelConfig.
eng-flavio-teixeira May 28, 2025
82d8e3a
- Refactor kernel configuration lists
eng-flavio-teixeira Jun 2, 2025
c739c15
- merge with
eng-flavio-teixeira Jun 2, 2025
d1f241d
- Fix issue with wgs / tpt / tpb in partial-pass kernel configuration.
eng-flavio-teixeira Jun 6, 2025
78dba8f
- Add validation and fix hardcoded offset calculation in partial pas…
eng-flavio-teixeira Jun 9, 2025
e232ad0
- Fixes for local transposition in SBCC partial-pass kernel generator.
eng-flavio-teixeira Jun 23, 2025
348e0a4
- Fix for hardcoded value in offset computation.
eng-flavio-teixeira Jun 24, 2025
a3be0ff
- Further fixes to calculate_offsets in partial-pass SBCC kernel.
eng-flavio-teixeira Jun 26, 2025
f52aefa
- clang format.
eng-flavio-teixeira Jun 26, 2025
3657b31
- Fixes for partial-pass SBCC kernel when using certain wgs and tpt …
eng-flavio-teixeira Jul 11, 2025
eee9bbd
- Fixes for partial-pass step_3_4 lds-to-reg and reg-to-lds generators.
eng-flavio-teixeira Jul 15, 2025
47b7020
- Merge with develop.
eng-flavio-teixeira Jul 15, 2025
25b4fa2
- Further resolved merge conflicts.
eng-flavio-teixeira Jul 15, 2025
a08e2d1
- Remove no longer needed lines from python config files.
eng-flavio-teixeira Jul 15, 2025
36608cd
- Further fixes to partial-pass SBCC calculate_offsets().
eng-flavio-teixeira Jul 17, 2025
3050aaa
- Fixes for partial-pass twiddle table generation.
eng-flavio-teixeira Jul 22, 2025
ab395af
- Add more partial-pass 3D lengths.
eng-flavio-teixeira Jul 23, 2025
3e40ba8
- Clean up.
eng-flavio-teixeira Jul 23, 2025
1061964
- Changes to reduce cost of local transpose in partial-pass steps 3-4.
eng-flavio-teixeira Jul 23, 2025
73d6b8c
- Add accuracy test coverage for new 3D lengths.
eng-flavio-teixeira Jul 23, 2025
28a3689
- Add more lengths to partial-pass perf suite.
eng-flavio-teixeira Jul 24, 2025
40bbe6f
- Merge with develop.
eng-flavio-teixeira Jul 24, 2025
280d78a
- Fix function pool init for partial-pass kernels.
eng-flavio-teixeira Jul 25, 2025
7cdcad5
- Further double-precision restrictions for some of the new partial-…
eng-flavio-teixeira Jul 25, 2025
5a06adb
- Clang format.
eng-flavio-teixeira Jul 29, 2025
a533919
- More clang format.
eng-flavio-teixeira Jul 29, 2025
9e691d9
Merge commit 'a5339195b470dd7138a54ef380e93d776900f32c' into import/d…
assistant-librarian[bot] Jul 31, 2025
163abf7
Merge branch 'develop' into import/develop/eng-flavio-teixeira_rocFFT…
eng-flavio-teixeira Jul 31, 2025
5bb79ce
- Changes to improve performance of local transposition.
eng-flavio-teixeira Jul 31, 2025
ea4e8f7
- Replace std::accumulate instances with arithmetic helper.
eng-flavio-teixeira Aug 1, 2025
e76ba66
- Address code review suggestions.
eng-flavio-teixeira Aug 1, 2025
3819868
- More code review suggestions.
eng-flavio-teixeira Aug 1, 2025
5205082
- Remove casts from offset calculation.
eng-flavio-teixeira Aug 1, 2025
719c7f8
- Add parent length to partial-pass kernel name.
eng-flavio-teixeira Aug 1, 2025
eef8ff7
Merge commit '5205082bed068e0f85a925cfe48dd4693694601f' into users/en…
eng-flavio-teixeira Aug 1, 2025
a567b61
- Merge with develop.
eng-flavio-teixeira Aug 1, 2025
d671758
- Add missing bound check in partial-pass kernel-generator python loop.
eng-flavio-teixeira Aug 6, 2025
575029a
- Update CHANGELOG
eng-flavio-teixeira Aug 7, 2025
625f542
Merge commit '575029add119a77719841afcb6c4bd4f1cfe0fc8' into rocFFT-p…
eng-flavio-teixeira Aug 8, 2025
5b9e3df
Merge commit 'd671758fbbf4bc2aab7b45bd848df002dd48f332' into rocFFT-p…
eng-flavio-teixeira Aug 8, 2025
5079d80
Merge branch 'users/eng-flavio-teixeira/partial_pass_local_transpose_…
eng-flavio-teixeira Aug 8, 2025
3c4d0fe
Merge branch 'users/eng-flavio-teixeira/fix_partial_pass_kernel_name'…
eng-flavio-teixeira Aug 8, 2025
ca813e2
Merge commit '43567c008e6763d51f8a91a75bb9dbd79ce5ce72' into rocFFT-p…
eng-flavio-teixeira Aug 15, 2025
fac4bfb
WIP
eng-flavio-teixeira Aug 15, 2025
9a0e58b
- Add threads_per_transform and transform_per_block configuration fo…
eng-flavio-teixeira Aug 25, 2025
bf260b3
- Add partial-pass work generator in stockham base.
eng-flavio-teixeira Aug 26, 2025
8491974
- Add further validation for pp_threads_per_transform.
eng-flavio-teixeira Aug 26, 2025
0f88967
- Move generate_pp_device_function from base to RR partial-pass spec…
eng-flavio-teixeira Aug 27, 2025
af2688f
- Add separate twiddle table for off dimension pass.
eng-flavio-teixeira Aug 28, 2025
1dd3f42
- Add generator for extra twiddle multiplication step in partial-pas…
eng-flavio-teixeira Sep 3, 2025
1504310
- Reduce number of index arithmetic operations in partial-pass gener…
eng-flavio-teixeira Sep 4, 2025
eed0ef4
Revert " - Reduce number of index arithmetic operations in partial-pa…
eng-flavio-teixeira Sep 5, 2025
fae3ee3
- Simplify full-pass device function generator in second kernel.
eng-flavio-teixeira Sep 22, 2025
68a1be8
- Remove no longer needed generators in second kernel.
eng-flavio-teixeira Sep 24, 2025
c543344
- Refactor second kernel generator names to follow first kernel nami…
eng-flavio-teixeira Sep 24, 2025
6a5a1a9
- More naming convention.
eng-flavio-teixeira Sep 24, 2025
24c4397
- Move add_work (partial-pass) generators to subclasses.
eng-flavio-teixeira Sep 25, 2025
d9e9278
- Refactor transform_per_block in second kernel.
eng-flavio-teixeira Sep 26, 2025
9e0b706
- Add partial-pass length
eng-flavio-teixeira Oct 15, 2025
75ff155
- Add more lengths.
eng-flavio-teixeira Oct 17, 2025
9773275
- Add more length
eng-flavio-teixeira Oct 17, 2025
f837814
- More length
eng-flavio-teixeira Oct 17, 2025
6311c86
- Add remaining lengths for md suite.
eng-flavio-teixeira Oct 20, 2025
048ed22
- Add new lengths to perf suite.
eng-flavio-teixeira Oct 20, 2025
5deff97
- Simplify partial-pass add work generator.
eng-flavio-teixeira Oct 22, 2025
a3654df
- Tweak wgs and tpt
eng-flavio-teixeira Oct 23, 2025
218573d
- Add ability to have per precision kernel configuration entries.
eng-flavio-teixeira Nov 3, 2025
e64821d
- Change bytes_per_element calculation to match what was previously …
eng-flavio-teixeira Nov 5, 2025
2512301
- Add support for arch specific entries in the function pool.
eng-flavio-teixeira Nov 24, 2025
15761c2
- Further changes for arch support in the function pool.
eng-flavio-teixeira Nov 27, 2025
91948d0
Merge commit '85ea1d36a78c3fca8485cbf6833497cfd18596ac' into function…
eng-flavio-teixeira Nov 27, 2025
7217193
- CHANGELOG.
eng-flavio-teixeira Nov 27, 2025
f99de06
Merge with develop
eng-flavio-teixeira Nov 28, 2025
e32b62a
- Resolve merge conflicts.
eng-flavio-teixeira Nov 28, 2025
326090e
- Remove suffix from get_curr_gcn_arch_name().
eng-flavio-teixeira Dec 1, 2025
7aad141
Merge commit '326090ef94a0c1bc1dbf20a19d1b59130fdc48f5' into rocFFT-p…
eng-flavio-teixeira Dec 1, 2025
8a9ee5b
- Address review suggestion.
eng-flavio-teixeira Dec 4, 2025
cfc455f
- Add partial-pass kernel entries per architecture.
eng-flavio-teixeira Dec 6, 2025
7b0d134
- Add missing gcn_arch_name in kernel_name()
eng-flavio-teixeira Dec 8, 2025
98bdeda
Merge commit '7b0d13412ca2f69581e6b176435770ae17ca0001' into rocFFT-p…
eng-flavio-teixeira Dec 8, 2025
7055d54
- Fix gcn_arch_name handling for partial-pass kernels.
eng-flavio-teixeira Dec 8, 2025
38c10a8
Merge commit '7055d54b172e264d172e11ab09883c1520303eed' into rocFFT-p…
eng-flavio-teixeira Dec 8, 2025
6a9fcd9
- Fix config entries for gfx1201
eng-flavio-teixeira Dec 8, 2025
554633e
- Fix partial-pass kernel check.
eng-flavio-teixeira Dec 9, 2025
1271ffe
Merge commit '554633e57ca2060cf1be9f62454d578bed864c35' into rocFFT-p…
eng-flavio-teixeira Dec 9, 2025
14feda5
- Refine batch cut-off for enabling partial pass kernels.
eng-flavio-teixeira Dec 9, 2025
b55afbd
- Add length condition to batch cut-off
eng-flavio-teixeira Dec 9, 2025
24db868
- Add comment.
eng-flavio-teixeira Dec 10, 2025
21ba23e
Merge branch 'develop' into function_pool_device_arch
eng-flavio-teixeira Dec 10, 2025
03fef0c
Merge branch 'develop' into function_pool_device_arch
eng-flavio-teixeira Dec 10, 2025
59676fb
Merge commit '03fef0cc8c8d94549674927d8e4cdb9fb6bd2c56' into rocFFT-p…
eng-flavio-teixeira Dec 10, 2025
ea12d45
- No need to put device arch name in StockhamGeneratorSpecs for aot …
eng-flavio-teixeira Dec 10, 2025
8467235
- Remove no longer needed calls to get_curr_gcn_arch_name to constru…
eng-flavio-teixeira Dec 10, 2025
a429b2b
- Remove header include.
eng-flavio-teixeira Dec 10, 2025
6114b2f
- Handle the case where FMKey is constructed without a visible HIP d…
eng-flavio-teixeira Dec 10, 2025
0980785
Revert " - No need to put device arch name in StockhamGeneratorSpecs …
eng-flavio-teixeira Dec 10, 2025
21e66ae
Revert " - Remove no longer needed calls to get_curr_gcn_arch_name to…
eng-flavio-teixeira Dec 10, 2025
f1e5984
Revert " - Remove header include."
eng-flavio-teixeira Dec 10, 2025
e22d8fa
Merge branch 'develop' into function_pool_device_arch
eng-flavio-teixeira Dec 10, 2025
36a30b7
- Take hip return into account.
eng-flavio-teixeira Dec 10, 2025
19ca6d6
Merge branch 'function_pool_device_arch' of https://github.com/eng-fl…
eng-flavio-teixeira Dec 10, 2025
76822b7
Merge commit '19ca6d6d66f1bd28c2635771c08f4a60290be6c8' into rocFFT-p…
eng-flavio-teixeira Dec 10, 2025
dd9653e
Merge commit '3748838c8bf212d47995dbda26f7b1cd8ae92ecb' into rocFFT-p…
eng-flavio-teixeira Dec 11, 2025
5f024ff
- Clang format.
eng-flavio-teixeira Dec 11, 2025
ace3e6d
- Python formatting.
eng-flavio-teixeira Dec 11, 2025
8b00d50
- CHANGELOG
eng-flavio-teixeira Dec 11, 2025
f9438c7
- Move SBCC's partial transform to its own device function.
eng-flavio-teixeira Dec 11, 2025
20674b1
- Update partial-pass perf suite dimension tag.
eng-flavio-teixeira Dec 12, 2025
b80f731
Merge branch 'develop' into rocFFT-pp_multiple_radices_sbcc
eng-flavio-teixeira Dec 12, 2025
6a8d479
Merge branch 'develop' into rocFFT-pp_multiple_radices_sbcc
eng-flavio-teixeira Dec 12, 2025
2956a25
- Handle the case where the AOT function pool process attemps to ca…
eng-flavio-teixeira Dec 18, 2025
4f08d8f
Merge branch 'rocFFT-pp_multiple_radices_sbcc' of https://github.com/…
eng-flavio-teixeira Dec 18, 2025
83a8045
Merge commit '786e4f8c027c85dd440b2b6375f6960fb92a25ef' into rocFFT-p…
eng-flavio-teixeira Dec 18, 2025
df0164b
- clang format
eng-flavio-teixeira Dec 18, 2025
2e1386e
Merge branch 'develop' into rocFFT-pp_multiple_radices_sbcc
eng-flavio-teixeira Dec 18, 2025
2c4af1e
- Rearrange 3d partial-pass config entries.
eng-flavio-teixeira Dec 18, 2025
1ed5f77
Merge branch 'rocFFT-pp_multiple_radices_sbcc' of https://github.com/…
eng-flavio-teixeira Dec 18, 2025
9e3f2c1
Merge commit '1ed5f771e719a19ab68f621655152c014653683b' into r2c_part…
eng-flavio-teixeira Jan 13, 2026
5b51fe1
WIP
eng-flavio-teixeira Jan 19, 2026
4351ef2
- WIP: Added changes to plan and generator to support r2c partial p…
eng-flavio-teixeira Feb 17, 2026
74f157c
- Fixes for partial-pass offset/strides calculation in real-to-compl…
eng-flavio-teixeira Feb 18, 2026
4cbf54b
- Merge with develop.
eng-flavio-teixeira Feb 18, 2026
66a4fb9
- Fix typo.
eng-flavio-teixeira Feb 18, 2026
219d0c9
- Resolve merge conflicts.
eng-flavio-teixeira Feb 18, 2026
76a3c84
Merge branch 'develop' into r2c_partial_pass
eng-flavio-teixeira Feb 18, 2026
e5d5e47
- Fixes to partial pass scheme selection logic in NodeFactory,
eng-flavio-teixeira Feb 25, 2026
3173a7f
- Python formatting.
eng-flavio-teixeira Mar 6, 2026
99f15df
- Handle case where partial-pass kernel config entry type is a list.
eng-flavio-teixeira Mar 6, 2026
a156dbb
- Merge partial-pass kernel config entries.
eng-flavio-teixeira Mar 7, 2026
6daa003
- Changes to node factory method to decide whether to use CS_REAL_3D…
eng-flavio-teixeira Mar 7, 2026
e9b631c
- Fix partial-pass perf suite.
eng-flavio-teixeira Mar 7, 2026
d228063
- Fix dist check in partial pass scheme decision.
eng-flavio-teixeira Mar 9, 2026
4c4229f
- Further fixes to check_pp_restrictions
eng-flavio-teixeira Mar 9, 2026
d8d832c
- Tweak batch cutoff for partial-pass r2c kernels.
eng-flavio-teixeira Mar 10, 2026
6b1c724
- Fix type in partial-pass restrictions check.
eng-flavio-teixeira Mar 12, 2026
884c989
- More batch cut-off tweaking.
eng-flavio-teixeira Mar 12, 2026
ca5f70b
- Re-arrange and merge partial-pass kernel config entries.
eng-flavio-teixeira Mar 12, 2026
a630ac6
- Simplify cutoff for enabling partial-pass.
eng-flavio-teixeira Mar 16, 2026
b64e4ae
- Merge with develop.
eng-flavio-teixeira Mar 16, 2026
95963c3
Fixes to kernel-generator.py
eng-flavio-teixeira Mar 16, 2026
b723ad7
- Tweak config_pp_3d
eng-flavio-teixeira Mar 16, 2026
f6d98d8
- Add low/high batch criteria to address gfx950 regression.
eng-flavio-teixeira Mar 17, 2026
98df418
- Resolve merge conflict.
eng-flavio-teixeira Mar 17, 2026
76dc526
- CHANGELOG.
eng-flavio-teixeira Mar 17, 2026
1873aab
Merge branch 'develop' into r2c_partial_pass
eng-flavio-teixeira Mar 18, 2026
bc9c0a6
- Fix typo.
eng-flavio-teixeira Mar 30, 2026
f82316e
- Rename batch cutoff values.
eng-flavio-teixeira Mar 30, 2026
2f6a5cf
- Resolve merge conflicts
eng-flavio-teixeira Apr 2, 2026
ff713c3
- clang format
eng-flavio-teixeira Apr 2, 2026
0777efb
Clean up Real3DPPNode::BuildTree_internal.
eng-flavio-teixeira Apr 2, 2026
5a29a75
- Address review comments.
eng-flavio-teixeira Apr 6, 2026
2315774
Remove no longer needed function from TreeNode.
eng-flavio-teixeira Apr 6, 2026
a420a7d
- Resolve merge conflict.
eng-flavio-teixeira Apr 6, 2026
1ad6274
Merge branch 'develop' into r2c_partial_pass
eng-flavio-teixeira Apr 7, 2026
a9ab6f0
- Add new partial-pass lengths to accuracy test suite.
eng-flavio-teixeira Apr 13, 2026
eed6f72
- Merge with develop.
eng-flavio-teixeira Apr 22, 2026
4ccd551
- Fix merge conflicts.
eng-flavio-teixeira Apr 22, 2026
97515f2
- Add new lengths to perf suite.
eng-flavio-teixeira Apr 22, 2026
e1e17e7
- Changes to support c2r partial pass in tree node.
eng-flavio-teixeira May 1, 2026
5d4307d
- Add stockham_pp_gen base class for partial kernels.
eng-flavio-teixeira May 6, 2026
292a506
Merge commit '5d4307de5221e7019ee3616b8a24d9758a1159fd' into c2r_part…
eng-flavio-teixeira May 7, 2026
ae2dc32
- Add support for the batch configuration directly in the python par…
eng-flavio-teixeira May 14, 2026
b976ff5
- Fix merge_kernel
eng-flavio-teixeira May 14, 2026
bcd5e82
- Fix print
eng-flavio-teixeira May 14, 2026
22dea61
Merge branch 'develop' into stockham_gen_pp_refactor
eng-flavio-teixeira May 14, 2026
72a4802
Merge commit 'bcd5e8221a0adc0332aabb55a1565c2a211811a3' into r2c_pp_n…
eng-flavio-teixeira May 14, 2026
390b60c
- Add new config entries.
eng-flavio-teixeira May 14, 2026
69e1a7d
- Tweak config values.
eng-flavio-teixeira May 16, 2026
95ae429
- fix typo.
eng-flavio-teixeira May 16, 2026
28f4cb2
- fix batch range for 100x100x100
eng-flavio-teixeira May 16, 2026
b1fc4c4
- Further tuning.
eng-flavio-teixeira May 19, 2026
54bfd41
- Improve comments.
eng-flavio-teixeira May 19, 2026
01adda1
- Remove generic entries that conflict with per-arch entries.
eng-flavio-teixeira May 20, 2026
4b15d4b
- tweak gfx1201
eng-flavio-teixeira May 20, 2026
4585bdc
Merge commit 'c158fde785a774b3670e444e6bef03fc9c05ce6e' into c2r_part…
eng-flavio-teixeira May 25, 2026
f32dab6
- Merge with pp_batch_kernel_config branch.
eng-flavio-teixeira May 25, 2026
b9bcfdd
Merge commit '4b15d4bcc8c5e1488bda11e1a2ebeab5195a6609' into c2r_part…
eng-flavio-teixeira May 25, 2026
f31ed8d
- Add missing r2c/c2r pre/post process generator function to RR part…
eng-flavio-teixeira May 25, 2026
59469e8
Merge commit 'f31ed8db2283ce08f129601840dd8175ccce5d15' into c2r_part…
eng-flavio-teixeira May 25, 2026
e160274
- remove debug code.
eng-flavio-teixeira May 25, 2026
8cd935f
Merge commit 'e160274529c5e5fefa3d7ca6d89b9757288500f4' into c2r_part…
eng-flavio-teixeira May 25, 2026
ed8e878
- Get root node transform type for specs struct.
eng-flavio-teixeira May 26, 2026
b704f2b
WIP
eng-flavio-teixeira Jun 22, 2026
b5a8c80
- Merge with develop
eng-flavio-teixeira Jun 22, 2026
e6196db
- Fix merge conflicts.
eng-flavio-teixeira Jun 22, 2026
9aed825
- Fix GetPPOffDim
eng-flavio-teixeira Jun 23, 2026
5c7837f
- clang format
eng-flavio-teixeira Jun 23, 2026
ef0d25e
Merge branch 'develop' into c2r_partial_pass
eng-flavio-teixeira Jul 20, 2026
7271394
WIP
eng-flavio-teixeira Jul 28, 2026
90f9e96
Merge branch 'c2r_partial_pass' of https://github.com/eng-flavio-teix…
eng-flavio-teixeira Jul 28, 2026
ad8d0f5
Merge commit '83298c0ec2dfa24fc0c8d5d70d5df888efb7003f' into c2r_part…
eng-flavio-teixeira Jul 28, 2026
897466b
- Fix issue with offset_pp not having the correct placement in the p…
eng-flavio-teixeira Sep 1, 2026
d812d6a
Merge commit 'e72781d41dc398ecbb88804184ea0450239caf5b' into c2r_part…
eng-flavio-teixeira Sep 2, 2026
dbc5b4b
- Merge with develop.
eng-flavio-teixeira Sep 24, 2026
a2d5e9f
- Extend generate_pp_steps_3_4_device_function to multiple passes.
eng-flavio-teixeira Sep 28, 2026
2d1bbf8
- Refactoring.
eng-flavio-teixeira Sep 28, 2026
1cbfb7c
- Add length 216x104x100 for c2r partial-pass.
eng-flavio-teixeira Sep 29, 2026
244af14
- Extend interleaved mode to work with factors_pp.size() > 1.
eng-flavio-teixeira Oct 1, 2026
a78bb89
Merge commit '17975eb833bbb6610db0a53adb8d56428656565a' into c2r_part…
eng-flavio-teixeira Oct 1, 2026
025163e
- CHANGELOG.
eng-flavio-teixeira Oct 1, 2026
44d1930
- Address review comments.
eng-flavio-teixeira Oct 6, 2026
2227886
- Address review comments.
eng-flavio-teixeira Oct 7, 2026
965ec25
- Add check for direct-to-reg.
eng-flavio-teixeira Oct 7, 2026
f9bb754
- copyright.
eng-flavio-teixeira Oct 7, 2026
b5d042a
- Address review comments.
eng-flavio-teixeira Oct 8, 2026
ab6f002
- Add comment to LDS column pattern enum.
eng-flavio-teixeira Oct 8, 2026
e480e93
- Implement review suggestions.
eng-flavio-teixeira Oct 8, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions projects/rocfft/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,11 @@ Documentation for rocFFT is available at
bricks used non-contiguous data layouts.
* Fixed a memory leak when `rocfft_plan_create` fails.

### Optimized

* Improved performance of unit-strided, interleaved, complex-to-real FFTs for the following lengths:
* (216,104,100)

## rocFFT 1.0.40 for ROCm 10.1

### Added
Expand Down
54 changes: 16 additions & 38 deletions projects/rocfft/library/src/device/generator/stockham_gen.cpp
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
// Copyright (C) 2021 - 2024 Advanced Micro Devices, Inc. All rights reserved.
// Copyright (C) 2021 - 2026 Advanced Micro Devices, Inc. All rights reserved.
//
// Permission is hereby granted, free of charge, to any person obtaining a copy
// of this software and associated documentation files (the "Software"), to deal
Expand Down Expand Up @@ -259,6 +259,16 @@ void output_json(const std::vector<GeneratedLauncher>& launchers,
output << "]";
}

// Give the params the node ordering an SBCC kernel generates against, which
// rotates the parent length so the SBCC's own transform dimension comes first.
// parent_length itself stays in plan order because it is the function-pool key.
static StockhamPartialPassParams pp_params_sbcc(StockhamPartialPassParams params)
{
std::rotate(
params.node_length.rbegin(), params.node_length.rbegin() + 1, params.node_length.rend());
return params;
}

// Render stockham partial-pass kernel generated launchers in JSON format.
void stockham_partial_pass_variants(const std::string& kernel_name,
const StockhamGeneratorSpecs& specs1,
Expand All @@ -269,7 +279,8 @@ void stockham_partial_pass_variants(const std::string& kernel_name
{
std::vector<GeneratedLauncher> launchers;

if(specs1.scheme == "CS_3D_PP" && specs2.scheme == "CS_3D_PP")
if((specs1.scheme == "CS_3D_PP" && specs2.scheme == "CS_3D_PP")
|| (specs1.scheme == "CS_REAL_3D_PP" && specs2.scheme == "CS_REAL_3D_PP"))
{
// SBRR_PP + SBCC_PP
if(params_1.current_dim == 0 && params_2.current_dim == 2)
Expand All @@ -288,7 +299,7 @@ void stockham_partial_pass_variants(const std::string& kernel_name
params_1.off_dim,
launchers);

StockhamPartialPassKernelCC kernelCC(specs2, params_2, false);
StockhamPartialPassKernelCC kernelCC(specs2, pp_params_sbcc(params_2), false);
make_launcher(specs2.precision,
*specs2.transform_type,
{{"pp_sbcc", specs2.scheme, "", ""}},
Expand Down Expand Up @@ -318,7 +329,7 @@ void stockham_partial_pass_variants(const std::string& kernel_name
params_1.off_dim,
launchers);

StockhamPartialPassKernelCC kernelRR(specs2, params_2, false);
StockhamPartialPassKernelCC kernelRR(specs2, pp_params_sbcc(params_2), false);
make_launcher(specs2.precision,
*specs2.transform_type,
{{"pp_stoc", specs2.scheme, "", ""}},
Expand Down Expand Up @@ -353,39 +364,6 @@ void stockham_partial_pass_variants(const std::string& kernel_name
throw std::runtime_error("invalid dimensions for CS_3D_PP");
}
}
else if(specs1.scheme == "CS_REAL_3D_PP" && specs2.scheme == "CS_REAL_3D_PP")
{
if(params_1.current_dim == 0 && params_2.current_dim == 2)
{
StockhamPartialPassKernelRR kernelRR(specs1, params_1);
make_launcher(specs1.precision,
*specs1.transform_type,
{{"pp_stoc", specs1.scheme, "", ""}},
kernelRR,
specs1.gcn_arch_name,
"CS_KERNEL_STOCKHAM_PP",
params_1.pp_threads_per_transform,
params_1.pp_factors_curr,
params_1.pp_factors_other,
params_1.current_dim,
params_1.off_dim,
launchers);

StockhamPartialPassKernelCC kernelCC(specs2, params_2, false);
make_launcher(specs2.precision,
*specs2.transform_type,
{{"pp_sbcc", specs2.scheme, "", ""}},
kernelCC,
specs2.gcn_arch_name,
"CS_KERNEL_STOCKHAM_PP_BLOCK_CC",
params_2.pp_threads_per_transform,
params_2.pp_factors_curr,
params_2.pp_factors_other,
params_2.current_dim,
params_2.off_dim,
launchers);
}
}
else
{
throw std::runtime_error("unhandled scheme");
Expand Down Expand Up @@ -622,7 +600,7 @@ void validate_pp_length(const std::string& scheme,

if(scheme == "CS_REAL_3D_PP" && pp_params.current_dim == 0)
{
// For real 3D FFTs, the z-dimension length is halved in the
// For r2c/c2r 3D FFTs, the x-dimension length is halved in the
// complex domain. So we need to multiply by 2 here to get
// the correct length to compare against parent_length.
length_curr *= 2;
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
// Copyright (C) 2021 - 2024 Advanced Micro Devices, Inc. All rights reserved.
// Copyright (C) 2021 - 2026 Advanced Micro Devices, Inc. All rights reserved.
//
// Permission is hereby granted, free of charge, to any person obtaining a copy
// of this software and associated documentation files (the "Software"), to deal
Expand Down Expand Up @@ -109,6 +109,7 @@ struct StockhamPartialPassParams
const std::vector<unsigned int>& pp_factors_curr,
const std::vector<unsigned int>& pp_factors_other)
: parent_length(parent_length)
, node_length(parent_length)
, pp_threads_per_transform(pp_threads_per_transform)
, current_dim(current_dim)
, off_dim(off_dim)
Expand All @@ -117,7 +118,13 @@ struct StockhamPartialPassParams
{
}

// Parent length in plan order. The launcher path emits this as the
// function-pool key, so it has to stay in plan order there.
std::vector<unsigned int> parent_length;
// The same lengths as the kernel's own node sees them: the SBRR's node is
// in plan order, the SBCC's is rotated so its transform dimension comes
// first. Code generation indexes this one, never parent_length.
std::vector<unsigned int> node_length;
unsigned int pp_threads_per_transform;
unsigned int current_dim = 0;
unsigned int off_dim = 0;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -578,7 +578,7 @@ struct StockhamKernel : public StockhamGeneratorSpecs
return stmts;
}

Function generate_lds_to_reg_input_function()
virtual Function generate_lds_to_reg_input_function()
{
std::string function_name = "lds_to_reg_input_length" + std::to_string(length) + "_device";

Expand Down Expand Up @@ -606,7 +606,7 @@ struct StockhamKernel : public StockhamGeneratorSpecs
return f;
}

Function generate_lds_from_reg_output_function()
virtual Function generate_lds_from_reg_output_function()
{
std::string function_name
= "lds_from_reg_output_length" + std::to_string(length) + "_device";
Expand Down Expand Up @@ -635,7 +635,7 @@ struct StockhamKernel : public StockhamGeneratorSpecs
return f;
}

Function generate_device_function()
virtual Function generate_device_function()
{
std::string function_name
= "forward_full_pass_length" + std::to_string(length) + "_" + tiling_name() + "_device";
Expand Down
Loading
Loading