[CI] Add Mistral Large 3 Eagle nightly performance test#14525
[CI] Add Mistral Large 3 Eagle nightly performance test#14525Kangyan-Zhou merged 7 commits intomainfrom
Conversation
Add nightly CI test for mistralai/Mistral-Large-3-675B-Instruct-2512-Eagle model. The test includes: - Benchmark test for batch sizes [1, 1, 8, 16, 64] - MGSM accuracy evaluation (threshold 0.90) Eagle-specific configuration: - --speculative-moe-runner-backend flashinfer_trtllm - --kv-cache-dtype auto (to avoid low AR with FP8 kv cache) - --attention-backend trtllm_mla - --tp 8
alisonshao
requested review from
Fridge003,
Kangyan-Zhou,
ispobock and
merrymercy
as code owners
Summary of ChangesHello @alisonshao, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request introduces a new continuous integration test for the Mistral Large 3 Eagle model. The test is designed to run nightly on an 8-GPU B200 system, ensuring ongoing performance benchmarking and accuracy evaluation (specifically MGSM) for this model with its optimized configurations. The primary goal is to maintain the stability and performance of the model within the system by catching regressions early. Highlights
Ignored Files
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds a new nightly performance and accuracy test for the mistralai/Mistral-Large-3-675B-Instruct-2512-Eagle model. The changes look good overall, introducing a new test file with benchmark and MGSM accuracy evaluation. I have a couple of suggestions to improve the robustness of environment variable handling and to clean up the test configuration.
|
|
||
| cls.model = MISTRAL_LARGE3_EAGLE_MODEL_PATH | ||
| cls.base_url = DEFAULT_URL_FOR_TEST | ||
| cls.batch_sizes = [1, 1, 8, 16, 64] |
There was a problem hiding this comment.
The batch_sizes list contains a duplicate value 1. This will cause the benchmark for a batch size of 1 to run twice, which seems unintentional and will consume extra CI time. If this is not intended for a specific reason like warm-up, please consider removing the duplicate.
| cls.batch_sizes = [1, 1, 8, 16, 64] | |
| cls.batch_sizes = [1, 8, 16, 64] |
| def tearDownClass(cls): | ||
| # Clean up environment variable | ||
| if "SGLANG_ENABLE_JIT_DEEPGEMM" in os.environ: | ||
| del os.environ["SGLANG_ENABLE_JIT_DEEPGEMM"] | ||
|
|
There was a problem hiding this comment.
The current method of cleaning up the environment variable by deleting it is not fully robust. If SGLANG_ENABLE_JIT_DEEPGEMM was set before these tests ran, its value would be lost, potentially affecting other tests running in the same process. A better practice is to save its original state in setUpClass and restore it in tearDownClass.
You can achieve this by:
- Adding
_original_jit_deepgemm = Noneat the class level. - In
setUpClass, saving the value:cls._original_jit_deepgemm = os.environ.get("SGLANG_ENABLE_JIT_DEEPGEMM")before setting the new value. - In
tearDownClass, restoring it:
if cls._original_jit_deepgemm is None:
os.environ.pop("SGLANG_ENABLE_JIT_DEEPGEMM", None)
else:
os.environ["SGLANG_ENABLE_JIT_DEEPGEMM"] = cls._original_jit_deepgemm
|
Should we add this model to h200 as well? |
The base Mistral Large 3 model (test_mistral_large3_basic.py) only works on B200 - it uses --attention-backend trtllm_mla which requires Blackwell (SM100) architecture. I tried running it on H200 previously but it didn't work. |
Move the Mistral Large 3 Eagle test into the same test file as the base Mistral Large 3 test, removing the need for a separate workflow step. This keeps the Eagle test within the existing nightly test job.
2 participants
Summary
mistralai/Mistral-Large-3-675B-Instruct-2512with Eagle speculative decodingtest_mistral_large3_perf.pyfile as a separate test classEagle-specific configuration
--speculative-algorithm EAGLE--speculative-draft-model-path mistralai/Mistral-Large-3-675B-Instruct-2512-Eagle--speculative-num-steps 3--speculative-eagle-topk 1--speculative-num-draft-tokens 4--kv-cache-dtype auto(to avoid low AR with FP8 kv cache)--attention-backend trtllm_mla--tp 8Related PRs