Expose cache_block_seq_len to API

[ispobock]

Motivation

As mentioned in #1195, cache_block_seq_len is an important parameter for performance. We should expose it to users.

Modification

Added argument --cache-block-seq-len for serve and chat api, and some benchmark scripts. The default value is 128, which is consistent with the previous internal setting.

[ispobock]

@lvhan028 @lzhangzz @AllentDan Could you help review this PR?

[zhyncs]

LGTM

[lvhan028]

@ispobock I am going to postpone this PR's review, since we are prioritizing #1211, which is very important in the upcoming release. This PR proposed a fallback strategy when turbomind engine hasn't supported some model but pytorch engine does.

Back to this PR, since both engines define the size of the k/v cache block, I think a common attribute name is better.
However, each engine has a different request on the value of the k/v cache block.
So we should work carefully on some cases, like turbomind engine fallback to pytorch engine

[ispobock]

@lvhan028 Sure. For Pytorch engine, there is a block_size argument with the default value 64. Currently it's not exposed to users. Before the change, here are some points to make sure:

1. Shall we also expose block_size for Pytorch engine?
2. Shall we use a common argument for both engines? If yes, which name we should take? block_size or cache_block_seq_len?
3. If fallback strategy is trigged:
   • If user set the argument value, maybe we can just use it
   • If not, maybe we need to change the default value from 128 to 64 for Pytorch

cc: @grimoire @RunningLeon @lzhangzz

[grimoire]

block_size might not be the given value in pytorch engine if lora adapters have been loaded. It will be adjusted to match the dimensions of adapters.

[ispobock]

block_size might not be the given value in pytorch engine if lora adapters have been loaded. It will be adjusted to match the dimensions of adapters.

@grimoire If user provide both adapter and block_size, maybe we need to ignore the block_size and log a warning to user.

[grimoire]

lmdeploy/lmdeploy/pytorch/engine/model_agent.py

Line 476 in 7dd97fd

logger.warning(f'infered block size: {block_size}')

This is the warning when block size is changed by engine.

[ispobock]

@lvhan028 It seems that block_size is a more common name for user, maybe we can use it for both engines.
If we use a common argument for both engines, it's not convenient to handle the different default values.
In #1195 (comment), we found the block_size = 64 is better than 128 for Turbomind engine. Shall we change the Turbomind default value to 64 to keep consistent with Pytorch engine?

[lvhan028]

I agree with the default value of 64.
But I prefer cache_block_seq_len since it goes with cache_max_entry_count and literally means the length of a sequence supported by a block.

[ispobock]

@lvhan028 I applied cache_block_seq_len argument for both engines and changed the default value to 64. The engine config mapping for cache_block_seq_len attribute in auto-backend is also handled. Pls help review.

[RunningLeon]

Better add more help info， eg. it should be multipliers of 32/64? and range?

[ispobock]

@RunningLeon @lvhan028 For Turbomind engine, it should be multipliers of 32 or 64 for different GPU compute capability version. For Pytorch engine, it maybe have different requirement. Do we need to specify the requirement here?

[ispobock]

More help message are added. @lvhan028 could you help review this PR?

[RunningLeon]

LGTM