update model_config for granite 4 models#821
Conversation
Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com>
|
👋 Hi! Thank you for contributing to vLLM support on Spyre. We also recommend installing prek and configuring it to check your code before every local commit. |
Signed-off-by: Joe Runde <joe@joerun.de>
joerunde
requested review from
nikolaospapandreou,
prashantgupta24,
rafvasq,
sducouedic,
tdoublep and
yannicks1
as code owners
| { | ||
| "architectures": [ | ||
| "GraniteMoeHybridForCausalLM" | ||
| "GraniteForCausalLM" |
There was a problem hiding this comment.
This is the diff between the old and new checkpoint configs
| "tie_word_embeddings": true, | ||
| "torch_dtype": "bfloat16", | ||
| "transformers_version": "4.56.0", | ||
| "transformers_version": "4.53.3", |
There was a problem hiding this comment.
Would different transformers version cause any issues between the 2 variants?
There was a problem hiding this comment.
I don't know the answer :/
It's probably not too relevant for us as we're not using transformers to load the model
| # This is really a dense model, but it has model type "granitemoehybrid" | ||
| # It has the same overrides as the regular dense variant |
There was a problem hiding this comment.
This makes me think that there might have be a mistake when creating the new checkpoint 🤔
There was a problem hiding this comment.
yeah it's kinda unclear how things evolved here, probably part of the reason why these configs haven't quite yet landed on hf hub
joerunde
left a comment
joerunde
left a comment
There was a problem hiding this comment.
Confirmed this loads new checkpoints correctly
3 participants
Description
Updates model configuration for Granite 4 dense models including for
granitevariant (instead ofgranitemoehybrid).Related Issues
Test Plan
Checklist
bash format.sh)Signed-off-by:line (DCO compliance)