[AutoParallel] Add auto parallel moe layer #9886
Conversation
|
Thanks for your contribution! |
Codecov ReportAttention: Patch coverage is
❌ Your patch check has failed because the patch coverage (14.25%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## develop #9886 +/- ##
===========================================
- Coverage 51.34% 51.17% -0.18%
===========================================
Files 745 748 +3
Lines 118567 119129 +562
===========================================
+ Hits 60877 60961 +84
- Misses 57690 58168 +478 ☔ View full report in Codecov by Sentry. |
| me = paddle.stack(me_list).mean(0) | ||
| ce = paddle.stack(ce_list).mean(0) | ||
| aux_loss = paddle.sum(me * ce) * float(self.num_experts) | ||
| return aux_loss |
There was a problem hiding this comment.
| # Make sure the capacity value does not exceed the number of tokens. | ||
| capacity = int(min(new_capacity, paddle.tensor(mask1.size(0)))) | ||
|
|
||
| l_aux = self._cal_aux_loss(gates, mask1) |
There was a problem hiding this comment.
pkuzyc
force-pushed
the
auto_moe_layer
branch
from
a75f03b to
8bfa877
Compare
PR types
New features
PR changes
Models
Description
Add auto parallel moe layer