Code

[Mixtral] Add mixtral moe#7803

Code

Merged

wawltor merged 7 commits intoPaddlePaddle:developPaddlePaddle/PaddleNLP:developfrom DesmonDay:add_moeDesmonDay/PaddleNLP:add_moeCopy head branch name to clipboardMar 1, 2024

+2,238-3

Commits

Commits on Jan 9, 2024

• Add mixtral moe

  DesmonDay

  committed
  abdc0da

  Copy full SHA for abdc0da

Commits on Feb 20, 2024

• Merge branch 'develop' of https://github.com/PaddlePaddle/PaddleNLP into add_moe

  DesmonDay

  committed
  b50e5f7

  Copy full SHA for b50e5f7

Commits on Feb 27, 2024

• fix mixtral moe

  DesmonDay

  committed
  1627c85

  Copy full SHA for 1627c85

Commits on Feb 28, 2024

• add mixtral lora config

  DesmonDay

  committed
  24103d1

  Copy full SHA for 24103d1

Commits on Feb 29, 2024

• fix load balancing loss

  DesmonDay

  committed
  d6768f1

  Copy full SHA for d6768f1

• add sft scripts

  DesmonDay

  committed
  bcb5aea

  Copy full SHA for bcb5aea

• add unittest

  DesmonDay

  committed
  fbe3b78

  Copy full SHA for fbe3b78