Which version is the comparison of the pressure measurement data provided in the project based on?

[dushulin]

The comparison data with TGI is based on what TGI version and startup parameters, as well as hardware.

[dushulin]

I loaded llama-7b in lightllm and tgi respectively, and found that the throughput of tgi-0.9.3 will be higher

[hiworldwzj]

@dushulin We tested on TGI==0.8.4, using the A800 80G graphics card. The latest TGI version has made many optimizations. However, for LightLLM performance testing, it requires a warm-up run because the Triton kernel needs to be compiled. When Triton releases version 2.1.0 officially, we will improve the performance of the decode attention operator, and we expect to achieve a performance improvement of 10% to 20% at that time.

[hiworldwzj]

@dushulin LightLLM aims to be the lightest pure Python inference library. In fact, we also have the version that combines TGI with LightLLM, which can achieve higher performance. At the same time, the current scheduling algorithm of TGI cannot handle scenarios with significant differences between long and short sentences very well. In such scenarios, LightLLM performs better in terms of performance. you can build a test set and try it。

[dushulin]

When testing lightllm, a 120s warmup is performed, the data will be better than non-warmup, but still lower than TGI-0.9.3

[hiworldwzj]

@dushulin Can you give more detail about this, We try it.