[AMD] Fix ZeroDivisionError in compute_normalized_prompt_logprobs

[sunxxuns]

Status Summary

PR #14721 (ZeroDivisionError fix):

• Lint failing (needs black formatting)
• URL: [AMD] Fix ZeroDivisionError in compute_normalized_prompt_logprobs #14721

PR #13760 (Diffusion AMD support):

• Some multimodal tests failing
• Latest commit: 5b4d2406: fix ci

I notice the attached file changes show reverting the AMD tolerance and runai_model_streamer fixes. Did you push those changes? That would explain why multimodal tests are now failing.

Would you like me to:

1. Fix the black formatting on PR [AMD] Fix ZeroDivisionError in compute_normalized_prompt_logprobs #14721?
2. Check what's causing the failures on PR [AMD] Enable all diffusion models and fix encoder loading on MI325 #13760?
3. Something else?

[gemini-code-assist]

Summary of Changes

Hello @sunxxuns, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the robustness of log probability calculations within the sglang backend. It specifically addresses potential ZeroDivisionError scenarios that could arise when processing token log probabilities, particularly during token healing and when computing normalized prompt logprobs. The changes ensure that the system gracefully handles cases where no valid log probabilities are available or when division by zero would otherwise occur.

Highlights

• Prevent ZeroDivisionError in compute_normalized_prompt_logprobs: The compute_normalized_prompt_logprobs function now explicitly checks if there are any valid log probabilities (values) before attempting to calculate their average. If no valid logprobs are found, it returns float("-inf") to avoid division by zero.
• Robustness in select function's token healing logic: The select function now includes checks to skip processing if input_token_logprobs are empty or invalid. Additionally, it ensures that division by zero is avoided when adjusting normalized_prompt_logprobs during token healing by checking if num_tokens is greater than 1 before performing the division.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request effectively addresses potential ZeroDivisionError issues in the runtime_endpoint.py file. The changes introduce robust checks to ensure that division operations are only performed when the denominator is not zero, preventing runtime crashes and improving the stability of the system. Additionally, an IndexError scenario is handled defensively. The added comments clearly explain the rationale behind the changes.

[gemini-code-assist]

This check is a good addition to prevent IndexError when input_token_logprobs[i] or input_token_logprobs[i][0] might be empty, which could occur with certain backend responses. It improves the robustness of the token healing logic.

[gemini-code-assist]

This modification correctly prevents a ZeroDivisionError if input_token_logprobs[i] contains only one token. By checking if num_tokens > 1 before performing the division, the code avoids a critical runtime error. This directly addresses the issue mentioned in the PR title.

[gemini-code-assist]

This is a crucial fix to prevent a ZeroDivisionError when values is an empty list. Returning float("-inf") is a sensible approach for log-probabilities in such cases, ensuring that this choice is not inadvertently selected in subsequent processing steps. This directly addresses the core problem of division by zero.

[sunxxuns]

Closing in favor of #14721 which has a cleaner diff (just the logprobs fix without the diffusion CI changes)