【AdvancedCompiler】Add te.swiglu&dswiglu #1062
Conversation
…TE.swiglu&dswiglu Integrate latest changes from remote branch to resolve divergence, enabling smooth push of local performance-related fixes for swiglu & dswiglu. Merge branch 'add-TE.swiglu&dswiglu' of https://github.com/AdvancedCompiler/FlagGems into add-TE.swiglu&dswiglu
|
|
|
|
plz sign CLA |
|
The original code seems to have fluctuations causing errors, which led to the failure of the special-op-test / container-unit-test (pull_request). We don't have the option to rerun it on our end.If convenient, please have the CI rerun this test separately. |
kiddyjinjin
changed the title
| ctx.quantizer = quantizer | ||
|
|
||
| shape = input_tensor.shape | ||
| H = shape[-1] // 2 |
There was a problem hiding this comment.
Is it necessary to add the assertion: assert H.shape[-1] % 2 == 0
There was a problem hiding this comment.
Assertions have currently been added in dswiglu (refer to the implementation in te). Since there is already checking code in swiglu, it is not necessary to repeat the assertions.
There was a problem hiding this comment.
May need to add more test cases to meet the test coverage rate requirements.
There was a problem hiding this comment.
The correctness test cases have been increased to 12, resulting in 72 outputs in total: 12 test cases × 3 dtypes × 2 (forward and backward).
7 participants
PR Category
Operator
Type of Change
New Feature
Description
Add transformer_engine_torch.swiglu&transformer_engine_torch.dswiglu
Issue
Progress
Performance