Refactor(prefix_cache): refactor init prefix cache, server args

[jimmy-evo]

Motivation

there are too many args and prefix cache implementation.

some of them are confilict.

too many if-else for disable_radix_cache and enable_hicache
and check between enable_lmcache and disable_radix_cache is forgot

use prefix_cache_backend and discard disable_radix_cache and enable_hicache

Modifications

introduce new arg: --prefix-cache-backend

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.

[gemini-code-assist]

Summary of Changes

Hello @jinmingyi1998, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly refactors how different prefix cache implementations are selected and configured within the system. By introducing a unified prefix-cache-backend argument, it streamlines the setup process, enhances configuration clarity, and prevents conflicts that arose from the previous fragmented argument structure.

Highlights

• Consolidated Prefix Cache Configuration: Replaced multiple conflicting boolean arguments like disable_radix_cache, enable_hicache, and enable_lmcache with a single, explicit --prefix-cache-backend argument. This simplifies configuration and reduces potential errors.
• Improved Code Clarity: Refactored the init_memory_pool_and_cache function in scheduler.py to use the new prefix_cache_backend enum, eliminating complex if-else chains and making the cache initialization logic more readable and maintainable.
• Deprecation of Old Arguments: The previous arguments (--disable-radix-cache, --enable-hierarchical-cache, --enable-lmcache) are now deprecated, with warnings issued if used, and their values are mapped to the new --prefix-cache-backend.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request refactors the prefix cache initialization by introducing a new --prefix-cache-backend argument. This simplifies the complex conditional logic for selecting a cache backend and deprecates several older, sometimes conflicting, arguments. The changes significantly improve code clarity and maintainability.

My review has identified one critical issue in the argument validation logic that could prevent the LoRA cache from being used. I've also included a suggestion to further improve the consistency of the refactoring for LoRA cache selection. Overall, this is a valuable refactoring.

[xiezhq-hermann]

I actually liked the changes here, please let me know your thoughts on this @hnyls2002 @merrymercy.
@jinmingyi1998 sorry for the slow response but can you help resolve the conflict and then we can proceed? The only problem might be a lot of the documents, blogs and CI tests have been using the old interfaces, not sure how long we should keep the deprecated interfaces for.

[jimmy-evo]

@xiezhq-hermann @hnyls2002
My goal is to optimize server configuration arguments.

Initially, simply disabling or enabling certain features and then deriving the final behavior required users to have a deep understanding of the code logic. This is especially challenging now that there are so many caching methods available.

for example

    prefix_cache_backend: Literal[
        "radix",
        "radix_cpp",
        "chunk",
        "hierarchical",
        "hierarchical_cpp",
        "lora",
        "lmcache",
        "swa_radix",
        "swa_chunk",
        "none",
    ] = "radix"

[hnyls2002]

@jinmingyi1998 No, we prefer using simply combinations of --enable-xxx instead of --prefix-cache-bakend. Some users would not know which exact cache they use.