[RFC]: Optimizing the Expand and Shrink Algorithms for LoRA Inference in Dense Models #28190
Description
Motivation.
During multi-LoRA performance testing on NVIDIA H800 GPUs, we identified performance bottlenecks in the original LoRA expand and shrink operators during dense model inference.
Proposed Change.
By optimizing the execution branches of the relevant algorithms, we aim to enhance the overall GPU execution concurrency and improve operational efficiency.We have identified an optimization strategy and plan to implement it within one week.
Feedback Period.
No response
CC List.
No response
Any Other Things.
No response
Before submitting a new issue...
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.