[New Model]: Grok 2

[mgoin]

The model to consider.

https://huggingface.co/xai-org/grok-2

The closest model vllm already supports.

Grok 1 has already been supported (https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/grok1.py) and it seems Grok 2 is a few modifications ontop of it - still listing its architecture as Grok1ForCausalLM in the config.json

You can reference this sglang PR that implements the changes to the Grok1 definition to support the new version sgl-project/sglang#9532

What's your difficulty of supporting the model you want?

I think it should be relatively trivial since it modifies an existing arch. It does seem the checkpoint is in a weird format, so not sure if conversion is needed there

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[igor-susic1]

@mgoin if this one is up for grabs I would tackle it

[wenchen76]

Hi @mgoin, I’m interested in contributing to this issue. @igor-susic1, perhaps we could collaborate on it together. Thanks!

[igor-susic1]

@wenchen76 Sure, why not. We can switch communication to vLLM slack once we get a green light from @mgoin.

[mgoin]

Please take it on!

[wenchen76]

Hi @mgoin, thanks!
Hi @igor-susic1, I plan to open a draft PR shortly covering the backbone changes for the new model. We can continue collaborating on Slack to refine the PR and handle testing. How does that sound?

[BranZhai]

also maybe i can help with this issue

[Crucifixion-Fxl]

I can help too～

[david6666666]

Any Update?

[igor-susic1]

@david6666666 I switched to the residual MoE and scaling RoPE but the testing is a bit slower I did not have time last few days.

[yuanjunchai]

@igor-susic1 Hi! Do you need any help? I am also willing to contribute to this issue.

[jiangwu300]

Are there any updates on this?