feat(cute_dsl/moe): re-enable use_cold_l2_cache in CuteDslMoEWrapper TuningConfig - #3384
Conversation
…Config Sets `use_cold_l2_cache=True` in the autotuner TuningConfig at `flashinfer/fused_moe/cute_dsl/tuner.py`, matching TRT-LLM's `CuteDslFusedMoENvfp4Runner.tuning_config`. Flushing L2 between profile iterations yields autotune measurements representative of production cold-cache conditions; without it, back-to-back iterations of the same tactic benefit from L2-hit reuse and bias the profile pass toward tactics that look fast during profiling but aren't faster in production. Cold-L2 was previously unset as a workaround for a latent reference cycle in `CuteDslMoEWrapper` (PR flashinfer-ai#3286 commit `640e32e7`) that would produce reproducible NaN in `test_wrapper_with_autotune` on B200 CI. PR flashinfer-ai#3340 fixed that cycle via a weakref trampoline; this commit follows up by re-enabling the flag. The corresponding comment block is rewritten to give a brief rationale for the now-enabled state. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughThe tuner configuration in ChangesCold L2 Cache Configuration
Estimated code review effort🎯 1 (Trivial) | ⏱️ ~3 minutes Possibly related PRs
Suggested labels
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request enables the use_cold_l2_cache parameter in the CuteDslFusedMoENvfp4Runner tuner. This change ensures that the L2 cache is flushed between profile iterations during autotuning, aligning with TRT-LLM's configuration to better simulate production cold-cache conditions. I have no feedback to provide.
|
/bot run |
4 participants
📌 Description
Sets
use_cold_l2_cache=Trueon the autotunerTuningConfiginflashinfer/fused_moe/cute_dsl/tuner.py, matching TRT-LLM'sCuteDslFusedMoENvfp4Runner.tuning_config. With cold-L2 ON, the autotuner flushes L2 between profile iterations and measures conservative timings, so the picked tactic is robustly fast under cold-cache conditions; without it, back-to-back iterations of the same tactic benefit from L2-hit reuse and bias the pick toward tactics that look fast during profiling but aren't faster in production.🔍 Related Issues
#3286
#3340
🚀 Pull Request Checklist
Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete.
✅ Pre-commit Checks
pre-commitby runningpip install pre-commit(or used your preferred method).pre-commit install.pre-commit run --all-filesand fixed any reported issues.🧪 Tests
unittest, etc.).Reviewer Notes
Summary by CodeRabbit