Add GraniteMoeHybrid support for 4.0#37658

Merged

ArthurZucker merged 79 commits intohuggingface:mainfrom

Ssukriti:granitemoe_hybrid_external_cleanup

May 6, 2025

Commits on Apr 30, 2025

initial config and MLA layer

Ssukriti
authored and
alex-jw-brooks
committed
first pass at decoder

Ssukriti
authored and
alex-jw-brooks
committed
completion of layers

Ssukriti
authored and
alex-jw-brooks
committed
modeling class

Ssukriti
authored and
alex-jw-brooks
committed
adding hybrid class to imports

Ssukriti
authored and
alex-jw-brooks
committed
fix imports granitemoehybrid

Ssukriti
authored and
alex-jw-brooks
committed
fix granitehybrid imports

Ssukriti
authored and
alex-jw-brooks
committed
fix granitehybrid import

Ssukriti
authored and
alex-jw-brooks
committed
fix generated modeling file

Ssukriti
authored and
alex-jw-brooks
committed
add some comments

Ssukriti
authored and
alex-jw-brooks
committed
minor fixes in layers

Ssukriti
authored and
alex-jw-brooks
committed
add sharedMLP layer

Ssukriti
authored and
alex-jw-brooks
committed
correct layer names

Ssukriti
authored and
alex-jw-brooks
committed
fixes in mamba config

Ssukriti
authored and
alex-jw-brooks
committed
fix mamba config

Ssukriti
authored and
alex-jw-brooks
committed
change name of MLP layer

Ssukriti
authored and
alex-jw-brooks
committed
fix seq mizer layers

Ssukriti
authored and
alex-jw-brooks
committed
correct mamba config

Ssukriti
authored and
alex-jw-brooks
committed
fixes in param names

Ssukriti
authored and
alex-jw-brooks
committed
enable hybrid model

Ssukriti
authored and
alex-jw-brooks
committed
update config

Ssukriti
authored and
alex-jw-brooks
committed
fix config granite hybrid

Ssukriti
authored and
alex-jw-brooks
committed
fix attention layer

Ssukriti
authored and
alex-jw-brooks
committed
cleanup to re-use mamba code

Ssukriti
authored and
alex-jw-brooks
committed
keep layer types

Ssukriti
authored and
alex-jw-brooks
committed
attention bias cleanup

Ssukriti
authored and
alex-jw-brooks
committed
update mamba layer name

Ssukriti
authored and
alex-jw-brooks
committed
first pass at tests

Ssukriti
authored and
alex-jw-brooks
committed
first pass at tests

Ssukriti
authored and
alex-jw-brooks
committed
use granite attention

Ssukriti
authored and
alex-jw-brooks
committed
fix: self attn weights

Ssukriti
authored and
alex-jw-brooks
committed
pass at making pos_emb optional

Ssukriti
authored and
alex-jw-brooks
committed
initialize self_attn only as needed

Ssukriti
authored and
alex-jw-brooks
committed
overwrite forward to create HybridMambaCache

Ssukriti
authored and
alex-jw-brooks
committed
Log invalid layer types
alex-jw-brooks
committed
Add attention outputs test
alex-jw-brooks
committed
Only emit attentions/logits if not None
alex-jw-brooks
committed
Fix config test hidden size divisibility
alex-jw-brooks
committed
mark granitmoehybrid as stateful
alex-jw-brooks
committed
Initialize mamba convolutional layers
alex-jw-brooks
committed
Formatting fixes
alex-jw-brooks
committed
config docstring, removed some unused attrs
alex-jw-brooks
committed
Fix missing arg in models test
alex-jw-brooks
committed
Fix create and check decoder model test
alex-jw-brooks
committed
support logits to keep in granitemoe
alex-jw-brooks
committed
regen to pass logits_to_keep
alex-jw-brooks
committed
Allow None or rope
alex-jw-brooks
committed
Fix gradient checkpointing
alex-jw-brooks
committed
Add granitemoehybrid as special cache for generate check
alex-jw-brooks
committed
Remove unused MLA refs
alex-jw-brooks
committed
Fix mamba layer mask
alex-jw-brooks
committed
Remove logits to keep from config
alex-jw-brooks
committed
Minor docstring nits
alex-jw-brooks
committed
Update licenses
alex-jw-brooks
committed
Enable cache by default
alex-jw-brooks
committed
map layer types to layer block type
alex-jw-brooks
committed
First pass at granite moe hybrid docs
alex-jw-brooks
committed
Ignore granite moe hybrid in valid checkpoint check
alex-jw-brooks
committed
Align attention interfaces
alex-jw-brooks
committed
regenerate modular granitemoeshared attention interface
alex-jw-brooks
committed
Align granite moe hybrid attn interface
alex-jw-brooks
committed
run formatting
alex-jw-brooks
committed
Handle mamba initialization
alex-jw-brooks
committed
avoid conditional attr defs
alex-jw-brooks
committed
Move hybrid layer validation to config
alex-jw-brooks
committed
Add placeholder integration tests
alex-jw-brooks
committed
Docs nits / Update model names
alex-jw-brooks
committed

Commits on May 2, 2025

Commits on May 5, 2025

link preview model in docs
alex-jw-brooks
committed