[Kernel] Split moe tuned configs

[jeejeelee]

Purpose

Part of #24112

Test Plan

Test Result

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[gemini-code-assist]

Code Review

This pull request refactors the MoE kernel configuration loading by introducing versioned directories for Triton. This is a good step towards better maintainability. I've identified a potential issue with the strict version matching for Triton configurations, which could lead to silent performance regressions. I've provided a suggestion to make the version matching more robust. The rest of the changes, including moving existing configs to a legacy_configs directory, look good.

[gemini-code-assist]

The current logic for selecting a Triton-versioned configuration is too strict. It checks for an exact version match (e.g., triton_3_4_0). If a user updates to a new patch release of Triton (e.g., 3.4.1), this check will fail, and vLLM will fall back to legacy or default configurations, which could cause a silent performance regression. This is a high-severity risk for a performance-critical library.

To make this more robust, I suggest implementing a fallback mechanism to use the latest compatible configuration. The suggested code below finds the latest available configuration version that is less than or equal to the current Triton version. It also improves logging by listing all checked paths if no configuration is found.

    def _check_config_file_path(path: str,
                                extra_info: str = ''
                                ) -> Optional[dict[int, Any]]:
        if os.path.exists(path):
            with open(path) as f:
                log_msg = f'Using configuration from {path} for MoE layer.'
                if extra_info:
                    log_msg += f' {extra_info}'
                logger.info(log_msg)
                return {int(key): val for key, val in json.load(f).items()}
        return None

    paths_checked = []

    # P1 User-specified configuration (highest priority)
    user_defined_config_folder = envs.VLLM_TUNED_CONFIG_FOLDER
    if user_defined_config_folder is not None:
        user_defined_config_file_path = os.path.join(
            user_defined_config_folder, json_file_name)
        paths_checked.append(user_defined_config_file_path)
        if val := _check_config_file_path(user_defined_config_file_path):
            return val

    # P2 Current Triton version configuration
    from packaging.version import parse as parse_version
    current_triton_version = parse_version(triton.__version__)
    configs_dir = os.path.join(os.path.dirname(os.path.realpath(__file__)),
                               'configs')

    available_versions = []
    if os.path.exists(configs_dir):
        for dirname in os.listdir(configs_dir):
            if dirname.startswith('triton_'):
                try:
                    version_str = dirname.replace('triton_', '').replace(
                        '_', '.')
                    available_versions.append(parse_version(version_str))
                except Exception:
                    # Ignore directories that don't match the version format
                    continue

    # Find the latest version that is not newer than the current triton version
    compatible_versions = sorted(
        [v for v in available_versions if v <= current_triton_version],
        reverse=True)

    for version in compatible_versions:
        version_name = f"triton_{str(version).replace('.', '_')}"
        config_path = os.path.join(configs_dir, version_name,
                                   json_file_name)
        paths_checked.append(config_path)
        if val := _check_config_file_path(config_path):
            return val

    # P3 Legacy configuration
    default_config_file_path = os.path.join(
        os.path.dirname(os.path.realpath(__file__)),
        'configs',
        'legacy_configs',
        json_file_name,
    )
    paths_checked.append(default_config_file_path)
    if val := _check_config_file_path(
            default_config_file_path,
            extra_info=
            'Loading config from the legacy configuration may be suboptimal, please update the corresponding config.',  # noqa: E501
    ):
        return val

    # If no optimized configuration is available, we will use the default
    # configuration
    logger.warning(
        ('Using default MoE config. Performance might be sub-optimal! '
         'Config file not found. Paths checked: %s'), paths_checked)
    return None

[youkaichao]

the idea looks good to me

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @jeejeelee.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork