LightLLM v1.0.0 Release!

[shihaobai]

New Features

• Cross-Process Request Object:

  • Retained and optimized the previous three-process architecture design.
  • Introduced a request object that can be accessed across processes, significantly reducing inter-process communication overhead.

• Folding of scheduling and model inference:

  • Implemented the folding of scheduling and model inference, significantly reducing communication overhead between the scheduler and modelrpc.

• CacheTensorManager:

  • New class to manage the allocation and release of Torch tensors within the framework.
  • Maximizes tensor sharing across layers at runtime and enhances memory sharing between different CUDA graphs.
  • On an 8x80GB H100 machine, using the DeepSeek-v2 model, LightLLM can run 200 CUDA graphs concurrently without out of memory (OOM).

• PD-Disaggregation Prototype

  • Dynamic registration of P and D nodes

• Fastest DeepSeek-R1 performance on H200

  • num_clients = 100. The input length of the test data is 1024, and the output follows a Gaussian distribution with a mean of 128
    -

For more details, stay tuned to our blog at https://www.light-ai.top/lightllm-blog/.

---

This discussion was created from the release LightLLM v1.0.0 Release!.