[Dispatch Creation] Create more multi-use dispatches #22011
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
IanWood1
force-pushed
the
enable_fuse_all_multi_use
branch
2 times, most recently
from
978db84 to
474562e
Compare
IanWood1
force-pushed
the
enable_fuse_all_multi_use
branch
from
9cecc04 to
ebfe3c7
Compare
IanWood1
force-pushed
the
enable_fuse_all_multi_use
branch
2 times, most recently
from
d5cf1a2 to
34db4ca
Compare
IanWood1
force-pushed
the
enable_fuse_all_multi_use
branch
from
34db4ca to
cbd1c3b
Compare
This was referenced Nov 6, 2025
IanWood1
force-pushed
the
enable_fuse_all_multi_use
branch
2 times, most recently
from
aba7406 to
ec408a4
Compare
IanWood1
force-pushed
the
enable_fuse_all_multi_use
branch
from
a827a1e to
d69930d
Compare
Contributor
Author
|
#22799 didn't seem to fix the issue. The reduction dispatch seems to get much slower when adding an additional result. Here is the before and after (slow) Update: I created #22841 and I'm just going to increase the golden times until that is resolved. |
IanWood1
force-pushed
the
enable_fuse_all_multi_use
branch
from
db02f0d to
e92c03f
Compare
Signed-off-by: Ian Wood <[email protected]>
Signed-off-by: Ian Wood <[email protected]>
Signed-off-by: Ian Wood <[email protected]>
Signed-off-by: Ian Wood <[email protected]>
Signed-off-by: Ian Wood <[email protected]>
Signed-off-by: Ian Wood <[email protected]>
Signed-off-by: Ian Wood <[email protected]>
Contributor
Author
|
Ignore the failure on PkgCI / Test Torch / torch_models tests :: amdgpu_mi325_gfx942 (pull_request), it's getting fixed with #22855 |
IanWood1
requested review from
Groverkss,
MaheshRavishankar and
kuhar
as code owners
MaheshRavishankar
approved these changes
Dec 8, 2025
Collaborator
MaheshRavishankar
left a comment
MaheshRavishankar
left a comment
There was a problem hiding this comment.
Long time coming!
IanWood1
added a commit
that referenced
this pull request
Dec 9, 2025
Lowers golden dispatch counts to reflect the expected numbers after #22011 ci-extra: test_torch Signed-off-by: Ian Wood <[email protected]>
keshavvinayak01
pushed a commit
that referenced
this pull request
Jan 27, 2026
This change allows producers to try to fuse with all consumers. Previously, fusing with multiple consumers was only allowed if the consumers were all truncate ops. This has been removed. This has some side effects that require a few other accompanying changes: 1. This PR can lead to dispatches with many ops and many operands to the dispatch. To prevent forming dispatches with more operands than the runtime can handle, `wouldExceedOperandLimit` was added to limit the number of operands to 16. 2. The golden times for datatiling llama decode were slightly increased. See #22841 for more details. 3. `options.numIterations = 32` was added to more aggressively fuse multi-use elementwise ops in the same dispatch to prevent codegen issues. 4. Changed check to prevent fusion from `IREE::LinalgExt::isBitExtendOp()` to `IREE::Flow::isClonableIntoDispatchOp()` to prevent fusing with scatter's index producer when it should be cloned. 5. Changed error to warning when `IREE::Flow::moveFollowingOpIntoDispatchRegion` fails. This can occur because `hasTransitiveDependencyOnFusionGroup` does not account for moving ops into dispatch regions. For example, A and B have no use-def relation and A does have a "transitive dep on the fusion group" but B doesn't. If you place A and B in the same dispatch, then asking if B `hasTransitiveDependencyOnFusionGroup` you must also consider all ops in the dispatch too. Which is currently unaccounted for. Side note: this change was supposed to be made in #22708, but I think I merged without actually making this change. Related: #22528 Closes: #22462 ci-extra: test_torch --------- Signed-off-by: Ian Wood <[email protected]> Signed-off-by: Keshav Vinayak Jha <[email protected]>
keshavvinayak01
pushed a commit
that referenced
this pull request
Jan 27, 2026
Lowers golden dispatch counts to reflect the expected numbers after #22011 ci-extra: test_torch Signed-off-by: Ian Wood <[email protected]> Signed-off-by: Keshav Vinayak Jha <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change allows producers to try to fuse with all consumers. Previously, fusing with multiple consumers was only allowed if the consumers were all truncate ops. This has been removed.
This has some side effects that require a few other accompanying changes:
wouldExceedOperandLimitwas added to limit the number of operands to 16.iree_encoding.set_encodingperformance #22841 for more details.options.numIterations = 32was added to more aggressively fuse multi-use elementwise ops in the same dispatch to prevent codegen issues.IREE::LinalgExt::isBitExtendOp()toIREE::Flow::isClonableIntoDispatchOp()to prevent fusing with scatter's index producer when it should be cloned.IREE::Flow::moveFollowingOpIntoDispatchRegionfails. This can occur becausehasTransitiveDependencyOnFusionGroupdoes not account for moving ops into dispatch regions. For example, A and B have no use-def relation and A does have a "transitive dep on the fusion group" but B doesn't. If you place A and B in the same dispatch, then asking if BhasTransitiveDependencyOnFusionGroupyou must also consider all ops in the dispatch too. Which is currently unaccounted for. Side note: this change was supposed to be made in [Dispatch Creation] Don't fuse uses from above #22708, but I think I merged without actually making this change.Related: #22528
Closes: #22462
ci-extra: test_torch