MoE Models: option to add load balancing loss#1765

Merged

kashif merged 7 commits intohuggingface:mainfrom

claralp:moe_add_load_balancing_loss

Jun 24, 2024

Commits on Jun 23, 2024

KTO: add aux loss
Clara Luise Pohland
committed
use router_aux_loss_coef in KtoTrainer when aux_loss enabled
Clara Luise Pohland
committed
align optional aux_loss in DPO, KTO, CPO, ORPO
Clara Luise Pohland
committed
precommit changes
Clara Luise Pohland
committed

Commits on Jun 24, 2024

fix KL forward kwargs
Clara Luise Pohland
committed
add aux_loss doku entry
Clara Luise Pohland
committed
apply docs suggestions
Clara Luise Pohland
committed