Support qwen3 next

[sufubao]

No description provided.

[gemini-code-assist]

Summary of Changes

Hello @sufubao, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the LightLLM framework by integrating the Qwen3Next model, which employs a sophisticated hybrid attention architecture. The changes involve a fundamental redesign of the memory management system to accommodate both traditional KV caches and new state buffers required by the linear attention layers. This includes the introduction of specialized weight and inference classes for Qwen3Next, backed by a comprehensive set of optimized Triton kernels to ensure efficient execution of its novel attention mechanisms. The overall aim is to broaden LightLLM's model compatibility and demonstrate support for cutting-edge, high-performance attention architectures.

Highlights

• Qwen3Next Model Integration: Introduces comprehensive support for the Qwen3Next model, featuring a novel hybrid attention architecture.
• Hybrid Attention Mechanism: Implements a Gated Delta Net (linear attention) for specific layers, alternating with traditional full attention based on a full_attention_interval configuration.
• Advanced Memory Management: A new Qwen3NextMemoryManager is introduced to efficiently handle both KV caches for full attention layers and specialized conv_state_buffers and ssm_state_buffers for linear attention layers.
• Standardized KV Buffer Access: Refactors KV buffer access across various existing models (Bloom, Deepseek2, Llama, Phi3) to use a new get_kv_buffer method, promoting a more controlled and flexible memory management interface.
• New Weight and Inference Classes: Adds ParameterWeight and TpParameterWeight for generic parameter handling, along with dedicated Qwen3NextTransformerLayerWeight and Qwen3NextTransformerLayerInfer classes to manage the unique layer structure and inference logic of Qwen3Next.
• Optimized Triton Kernels: Incorporates a suite of new Triton kernels for high-performance execution of linear attention components, including causal convolution, gated delta rule, fused recurrent operations, L2 normalization, and gated RMSNorm.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Summary of Changes

Hello @sufubao, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly expands the model capabilities by integrating the Qwen3Next model. This integration involves a sophisticated hybrid attention mechanism, requiring a complete overhaul of memory management to support both traditional KV caching and novel Mamba-like state buffers. The changes also introduce a suite of optimized Triton kernels to ensure high performance for the new linear attention components, alongside general improvements to memory access patterns across existing models.

Highlights

• Qwen3Next Model Support: Introduced comprehensive support for the Qwen3Next model, including its unique hybrid attention architecture that combines traditional full attention with a novel linear attention mechanism (Gated Delta Net).
• Memory Management Refactor: Implemented a specialized Qwen3NextMemoryManager to efficiently handle both the standard KV cache for full attention layers and new Mamba-like state buffers (convolutional and SSM states) required by the linear attention layers.
• Generalized KV Buffer Access: Refactored KV cache access across various model implementations (Bloom, Deepseek2, Llama, Phi3, and common templates) to use a new get_kv_buffer() method, promoting a more abstract and consistent interface for memory management.
• New Triton Kernels for Linear Attention: Added several highly optimized Triton kernels to accelerate the Gated Delta Net operations, including causal 1D convolutions, chunked and fused recurrent gated delta rules, and gated RMSNorm.
• Enhanced RMSNorm Implementation: The Qwen3NextTransformerLayerInfer now features a custom 'Zero-Centered RMSNorm' implementation, and the Qwen3MOETransformerLayerInfer (base for Qwen3Next) has been updated to use a new rmsnorm method and incorporate partial_rotary_factor.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request adds support for the qwen3next model, which appears to be a hybrid architecture combining standard attention with a Mamba-like linear attention mechanism. The implementation is well-structured across several new files. A significant part of this PR is also a beneficial refactoring that encapsulates direct access to the kv_buffer within the MemoryManager by introducing a get_kv_buffer method, improving code maintainability across several model implementations. I've identified a bug in the rmsnorm implementation for qwen3_moe and an inconsistency in its application, along with a minor comment style issue in the new qwen3next code. Overall, the changes are substantial and well-organized.