[Feat][Router]: add extra support for YAML config file

[antoineauger]

As discussed in #506 (comment), this PR adds support for a YAML config file.

See the updated documentation (or below) for an example of a YAML config file. The content of the YAML config has been aligned with the JSON one for consistency.

service_discovery: static
routing_logic: roundrobin
callbacks: module.custom.callback_handler
static_models:
    facebook/opt-125m:
        static_backends:
            - http://localhost:9001
            - http://localhost:9003
        static_model_type: completion
    meta-llama/Llama-3.1-8B-Instruct:
        static_backends:
            - http://localhost:9002
        static_model_type: chat

FIX #506

🛠️ with ❤️ by Siemens

---

• Make sure the code changes pass the pre-commit checks.
• Sign-off your commit by using -s when doing git commit
• Try to classify PRs for easy understanding of the type of changes, such as [Bugfix], [Feat], and [CI].

Detailed Checklist (Click to Expand)

Thank you for your contribution to production-stack! Before submitting the pull request, please ensure the PR meets the following criteria. This helps us maintain the code quality and improve the efficiency of the review process.

PR Title and Classification

Please try to classify PRs for easy understanding of the type of changes. The PR title is prefixed appropriately to indicate the type of change. Please use one of the following:

• [Bugfix] for bug fixes.
• [CI/Build] for build or continuous integration improvements.
• [Doc] for documentation fixes and improvements.
• [Feat] for new features in the cluster (e.g., autoscaling, disaggregated prefill, etc.).
• [Router] for changes to the vllm_router (e.g., routing algorithm, router observability, etc.).
• [Misc] for PRs that do not fit the above categories. Please use this sparingly.

Note: If the PR spans more than one category, please include all relevant prefixes.

Code Quality

The PR need to meet the following code quality standards:

• Pass all linter checks. Please use pre-commit to format your code. See README.md for installation.
• The code need to be well-documented to ensure future contributors can easily understand the code.
• Please include sufficient tests to ensure the change is stay correct and robust. This includes both unit tests and integration tests.

DCO and Signed-off-by

When contributing changes to this project, you must agree to the DCO. Commits must include a Signed-off-by: header which certifies agreement with the terms of the DCO.

Using -s with git commit will automatically add this header.

What to Expect for the Reviews

We aim to address all PRs in a timely manner. If no one reviews your PR within 5 days, please @-mention one of YuhanLiu11
, Shaoting-Feng or ApostaC.

[gemini-code-assist]

Summary of Changes

Hello @antoineauger, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the router's configuration flexibility by introducing comprehensive support for YAML configuration files. This allows users to define dynamic router settings, including service discovery, routing logic, and static model configurations, using the YAML format, which can be particularly beneficial for managing complex configurations. The changes involve updates to argument parsing, config loading logic, and the addition of a new utility module for YAML-specific processing.

Highlights

• YAML Configuration Support: Added the ability to load dynamic router configurations from YAML files via a new --dynamic-config-yaml command-line argument, providing an alternative to the existing JSON configuration.
• Unified Config Loading and Parsing: Refactored the configuration loading mechanism to support both YAML and JSON formats, ensuring mutual exclusivity between the two config file options. New utility functions were implemented to parse structured YAML data (e.g., static_models and static_aliases with nested details) and convert them into the flat string formats expected by the router's internal arguments.
• Documentation and Testing Updates: The README.md has been updated to reflect the new YAML configuration option, provide an example YAML config, and clarify the mutually exclusive nature of YAML and JSON config file arguments. Corresponding unit tests were added to validate the correct loading and processing of YAML configurations.
• Dependency Addition: The pyyaml library has been included as a new project dependency to enable YAML parsing capabilities.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments or fill out our survey to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This PR adds valuable support for YAML configuration files, enhancing usability. The implementation correctly makes YAML and JSON configurations mutually exclusive.

I've identified a critical issue with a duplicated dataclass attribute that must be addressed. Additionally, the YAML example in the README is misleading due to the use of duplicate keys, which will lead to incorrect configurations. I've also provided suggestions to refactor the new YAML utility functions for better performance and adherence to Python idioms.

Overall, a great addition with a few key areas for improvement.

[antoineauger]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces support for YAML configuration files for dynamic router configuration. I have identified a critical issue in the new YAML utility module that could cause the application to terminate unexpectedly, a high-severity bug where empty YAML files are not handled correctly, and a couple of medium-severity suggestions to improve code clarity and error handling.

[antoineauger]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces support for YAML configuration files, which enhances user flexibility. The implementation includes updates to handle both YAML and JSON formats, with corresponding documentation and tests. Review comments address an unused import and a potential issue with inconsistent static_model_type definitions in the YAML configuration.

[antoineauger]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces support for YAML configuration files, which is a great addition for user-friendliness. The implementation is solid, with good separation of concerns in a new yaml_utils.py file, updated documentation, and new tests. The use of a mutually exclusive group for the YAML and JSON config file arguments is a nice touch. I have one suggestion to improve the robustness of the YAML parsing by adding type validation. Otherwise, the changes look good to me.

[antoineauger]

@zerofishnoodles @YuhanLiu11 I did some review rounds with the AI, nice experience overall 🤖

Could you please take a look at this PR? Better to review it commit-by-commit, thanks!

/cc @max-wittig @nejch

[zerofishnoodles]

This feature looks super cool! Thanks for contributing! Can you fix the CI check issue, and then I think it is good to go.

[antoineauger]

@zerofishnoodles I fixed the CI check in e551f4d and also fixed a bug related to the static_aliases that I discovered while doing some more local tests (see 5b2581a)

[zerofishnoodles]

LGTM

[YuhanLiu11]

@antoineauger Can you merge the latest changes from main?

[antoineauger]

@antoineauger Can you merge the latest changes from main?

@YuhanLiu11 Sure, done!