[BugFix] Fix cache issue in compilation_config

[BoyuanFeng]

#22204 added get_cached_compilation_config() with @lru_cache(maxsize=1) to cache compilation config access via

@lru_cache(maxsize=1)
def get_cached_compilation_config():
    """Cache config to avoid repeated calls to get_current_vllm_config()"""
    return get_current_vllm_config().compilation_config

Since get_cached_compilation_config() has no argument, it would never be refreshed, unless we explicitly call get_cached_compilation_config.cache_clear().

To refresh the config, #22204 implements get_cached_compilation_config.cache_clear() in set_current_vllm_config.

@contextmanager
def set_current_vllm_config(vllm_config: VllmConfig, check_compile=False, prefix: Optional[str] = None):
    try:
        _current_vllm_config = vllm_config
        _current_prefix = prefix
        yield
   finally:
        _current_vllm_config = old_vllm_config
        _current_prefix = old_prefix
        # Clear the compilation config cache when context changes
        get_cached_compilation_config.cache_clear() # <----- explicitly clear the cache

However, a bug would happen if an old_vllm_config has been accessed and cached before set_current_vllm_config(new_config) is called. In other words

config = old_config
with set_current_vllm_config(new_config):
    # old_config is accessed instead of new_config

This PR fixes the issue by clearing the cache when entering the context manager.

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.

[gemini-code-assist]

Code Review

This pull request addresses a caching bug in get_cached_compilation_config by ensuring the cache is cleared upon entering the set_current_vllm_config context manager. This prevents an old, cached configuration from being used when a new configuration is set. The fix is correct and directly solves the issue described. The added code comment clearly explains the necessity of this change. The implementation is sound, and I have no further recommendations.

[yewentao256]

LGTM, thanks for the work!

Is there any specific issue this PR could solve? (If there is a case that without this PR would cause trouble?)

[BoyuanFeng]

@yewentao256 I added a unit test test_cached_compilation_config to show the issue.
tldr: prior to this pr, if I call old_config = get_cached_compilation_config first, and then call with set_current_vllm_config(vllm_config):, the old_config.compilation_config is used within the context manager instead of the new vllm_config.compilation_config.

[yewentao256]

LGTM, thanks for the work!

[yewentao256]

Merge this PR as all CI tests pass