Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Mixture of Experts #479

Closed
sdtblck opened this issue Dec 12, 2021 · 0 comments · Fixed by #663 or #1129 · May be fixed by yang/gpt-neox#4
Closed

Add Mixture of Experts #479

sdtblck opened this issue Dec 12, 2021 · 0 comments · Fixed by #663 or #1129 · May be fixed by yang/gpt-neox#4
Assignees
@Quentin-Anthony
Labels
feature request New feature or request

Comments

@sdtblck
Copy link
Contributor

sdtblck commented Dec 12, 2021

from DeepSpeed-MoE for NLG: Reducing the training cost of language models by 5 times .

It should be a fairly simple addition as the codebase they open source is largely similar to ours (same base model, although we have diverged a bit since).
@sdtblck sdtblck added the feature request New feature or request label Dec 12, 2021
@Quentin-Anthony Quentin-Anthony self-assigned this Feb 10, 2022
@StellaAthena StellaAthena mentioned this issue Feb 25, 2022
Open
@StellaAthena StellaAthena linked a pull request Sep 18, 2022 that will close this issue
Latest DeepSpeed Support #663
Merged
5 tasks
yang added a commit to yang/gpt-neox that referenced this issue Jan 23, 2024
@yang
Add MoE
54d8df4 
Closes EleutherAI#479
@yang yang mentioned this issue Jan 23, 2024
Merged
yang added a commit to yang/gpt-neox that referenced this issue Jan 25, 2024
@yang
Add MoE
a743484 
Closes EleutherAI#479
yang added a commit to yang/gpt-neox that referenced this issue Jan 25, 2024
@yang
Add MoE
d6da58d 
Closes EleutherAI#479
yang added a commit to yang/gpt-neox that referenced this issue Jan 29, 2024
@yang
Add MoE
bf00aeb 
Closes EleutherAI#479
yang added a commit to yang/gpt-neox that referenced this issue Jan 30, 2024
@yang
Add MoE
776be24 
Closes EleutherAI#479
yang added a commit to yang/gpt-neox that referenced this issue Jan 30, 2024
@yang
Add MoE
0e5be89 
Thanks to dayofthepenguin for extensive testing

Closes EleutherAI#479
yang added a commit to yang/gpt-neox that referenced this issue Jan 30, 2024
@yang
Add MoE
2ecfd1e 
Thanks to dayofthepenguin for extensive testing

Closes EleutherAI#479
yang added a commit to yang/gpt-neox that referenced this issue Jan 30, 2024
@yang
Add DeepSpeed MoE
03c2cb8 
Thanks to dayofthepenguin for extensive testing

Closes EleutherAI#479
yang added a commit to yang/gpt-neox that referenced this issue Jan 30, 2024
@yang
Add DeepSpeed MoE
4362683 
Thanks to dayofthepenguin for extensive testing

Closes EleutherAI#479
yang added a commit to yang/gpt-neox that referenced this issue Feb 25, 2024
@yang
Add DeepSpeed MoE
4c8849d 
Thanks to dayofthepenguin for extensive testing

Closes EleutherAI#479
yang added a commit to yang/gpt-neox that referenced this issue Mar 4, 2024
@yang
Add DeepSpeed MoE
6f95a29 
Thanks to dayofthepenguin for extensive testing

Closes EleutherAI#479
yang added a commit to yang/gpt-neox that referenced this issue Mar 4, 2024
@yang
Add DeepSpeed MoE
bab8e4f 
Thanks to dayofthepenguin for extensive testing

Closes EleutherAI#479
@yang yang mentioned this issue Mar 4, 2024
Open
yang added a commit to yang/gpt-neox that referenced this issue Mar 6, 2024
@yang
Add DeepSpeed MoE
da69326 
Thanks to dayofthepenguin for extensive testing

Closes EleutherAI#479
yang added a commit to yang/gpt-neox that referenced this issue Mar 6, 2024
@yang
Add DeepSpeed MoE
89e16e8 
Thanks to dayofthepenguin for extensive testing

Closes EleutherAI#479
yang added a commit to yang/gpt-neox that referenced this issue Mar 6, 2024
@yang
Add DeepSpeed MoE
4825f1a 
Thanks to dayofthepenguin for extensive testing

Closes EleutherAI#479
yang added a commit to yang/gpt-neox that referenced this issue Mar 6, 2024
@yang
Add DeepSpeed MoE
60ba057 
Thanks to dayofthepenguin for extensive testing

Closes EleutherAI#479
Quentin-Anthony added a commit that referenced this issue Mar 7, 2024
@yang @Quentin-Anthony
Add MoE (#1129)
86758c3 
* Add DeepSpeed MoE

Thanks to dayofthepenguin for extensive testing

Closes #479

* Update NeoXArgs docs automatically

* pre-commit

* Update NeoXArgs docs automatically

---------

Co-authored-by: Yang Zhang <[email protected]>
Co-authored-by: github-actions <[email protected]>
Co-authored-by: Quentin Anthony <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Quentin-Anthony Quentin-Anthony

Labels
feature request New feature or request
Projects
None yet
Milestone
No milestone
2 participants
@Quentin-Anthony @sdtblck