[CI] Small Accuracy Eval Test for Deepseek Model

[yewentao256]

Purpose

Partially fix #23354

And we also have #24119 comments saying about unit test, so we add a small eval test here.

This may avoid some of the accuracy issues

CI test we reuse the LM Eval Small Models

Test

pytest -s -v tests/evals/gsm8k/test_gsm8k_correctness.py --config-list-file=tests/evals/gsm8k/configs/models-small.txt --tp-size=1 -k DeepSeek-V2-Lite-Chat

Evaluating: 100%|████████████████████████████████████████████████| 100/100 [00:10<00:00,  9.66it/s]
GSM8K Results for deepseek-ai/DeepSeek-V2-Lite-Chat:
  Accuracy: 0.660
  Expected: 0.300
  Questions: 100
  Invalid rate: 0.000
  Latency: 10.4s
  QPS: 9.7
✅ GSM8K test passed for deepseek-ai/DeepSeek-V2-Lite-Chat

[yewentao256]

@WoosukKwon @houseroad CC

[gemini-code-assist]

Code Review

This pull request adds a new manual accuracy evaluation test for the Deepseek V2 Lite model on the GSM8K benchmark. The changes include a new Buildkite CI step, a model-specific configuration file, and an update to the list of models to be tested. My review found a critical issue in the new CI step configuration where file paths appear to be incorrect, which would likely cause the test to fail. I've provided a suggestion to correct the paths.

[yewentao256]

@mgoin CC

[mgoin]

Might as well use a compressed version of the checkpoint to test multiple things at once and save time
What about https://huggingface.co/RedHatAI/DeepSeek-Coder-V2-Lite-Instruct-FP8?

[yewentao256]

Might as well use a compressed version of the checkpoint to test multiple things at once and save time What about https://huggingface.co/RedHatAI/DeepSeek-Coder-V2-Lite-Instruct-FP8?

@mgoin Updated, thanks for the suggestions!