docs: refactor speculative decoding doc

[alphabetc1]

Motivation

refactor speculative decoding doc

Modifications

Accuracy Tests

Benchmarking and Profiling

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review Process

1. Ping Merge Oncalls to start the PR flow. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • /tag-run-ci-label, /rerun-failed-ci, /tag-and-rerun-ci
4. After green CI and required approvals, ask Merge Oncalls to merge.

[gemini-code-assist]

Summary of Changes

Hello @alphabetc1, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request refactors the speculative decoding documentation to enhance clarity and provide updated information. The changes include more detailed explanations for torch.compile integration, the introduction of a new training framework for EAGLE-3, and several minor textual and formatting improvements. The overall goal is to make the documentation more precise and helpful for users implementing various speculative decoding strategies.

Highlights

• Documentation Clarity for torch.compile: Clarified the benefits and considerations for using torch.compile with EAGLE-2, emphasizing that performance gains can vary based on hardware and model configuration, and recommending benchmarking.
• EAGLE-3 Training Framework: Updated the documentation to include details about SpecForge, a dedicated training framework for EAGLE-3 speculative decoding models, providing relevant links to its repository, documentation, and blog post.
• Textual and Formatting Improvements: Made minor textual corrections, capitalization fixes, and formatting adjustments across the speculative decoding document for improved readability and consistency.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• docs/advanced_features/speculative_decoding.md
  • Updated the 'Notes / constraints' for 'EAGLE-2 + torch.compile' to clarify that benefits vary and benchmarking is recommended.
  • Changed 'Best throughput in the benchmark above' to 'Best throughput in the benchmark below' for EAGLE-3.
  • Capitalized 'Decoding' in the 'EAGLE-2 decoding' heading and its introductory sentence.
  • Expanded the explanation for torch.compile with EAGLE-2, detailing kernel-level optimizations and factors affecting speedup.
  • Corrected 'Eagle' to 'EAGLE' and 'token' to 'tokens' in the 'EAGLE-2 Decoding via Frequency-Ranked Speculative Sampling' section.
  • Added a space in 'MTP(Multi-Token Prediction)' to 'MTP (Multi-Token Prediction)'.
  • Adjusted the phrasing for --speculative-eagle-topk to include parentheses for clarity.
  • Added a new paragraph introducing SpecForge as a training framework for EAGLE-3 models, including links to its documentation and blog post.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[alphabetc1]

@b8zhong PTAL
In my tests (H100 + Llama/Qwen), torch.compile showed no benefits. I'm unsure if it helps other hardware/models.

[gemini-code-assist]

Code Review

The pull request refactors the speculative decoding documentation to improve clarity, fix minor typos, and update references to benchmarks and training frameworks. Key improvements include a more detailed explanation of the benefits and limitations of torch.compile, corrected benchmark references for EAGLE-3, and the addition of SpecForge as a training framework for EAGLE-3 models. The changes enhance the overall maintainability and accuracy of the documentation.

[b8zhong]

Thanks. @alphabetc1 I forgot to mention this one last thing: could we remove steps 5 and 6? Firstly, f16 is the same footprint as bf16, and I think the draft and target dtype need to be matched, so we don't really need this tips. Also I'm not sure if 5 exactly works (let me know if you found otherwise).

[alphabetc1]

Thanks. @alphabetc1 I forgot to mention this one last thing: could we remove steps 5 and 6? Firstly, f16 is the same footprint as bf16, and I think the draft and target dtype need to be matched, so we don't really need this tips. Also I'm not sure if 5 exactly works (let me know if you found otherwise).

done.PTLA

[b8zhong]

Thanks! @alphabetc1 Appreciate your work.