[do not merge] diffusion: pr ci consistency by yhyang201 · Pull Request #20139 · sgl-project/sglang

yhyang201 · 2026-03-08T19:25:19Z

Motivation

Modifications

Accuracy Tests

Benchmarking and Profiling

Checklist

Format your code according to the Format code with pre-commit.
Add unit tests according to the Run and add unit tests.
Update documentation according to Write documentations.
Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
Follow the SGLang code style guidance.

Review Process

Ping Merge Oncalls to start the PR flow. See the PR Merge Process.
Get approvals from CODEOWNERS and other reviewers.
Trigger CI tests with comments or contact authorized users to do so.
- /tag-run-ci-label, /rerun-failed-ci, /tag-and-rerun-ci
After green CI and required approvals, ask Merge Oncalls to merge.

- Add consistency_threshold.json for per-case threshold config - Add CLIP-based consistency checking utilities: get_clip_model, compute_clip_embedding, compute_clip_similarity, compare_with_gt, load_gt_embeddings, gt_exists, get_clip_threshold, ConsistencyResult - Add _validate_consistency() method to test_server_common.py - Call consistency validation after performance validation in test_diffusion_generation

gemini-code-assist · 2026-03-08T19:25:37Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the CI pipeline for diffusion models by introducing a robust consistency testing framework. It ensures that generated outputs (images and videos) remain consistent across different runs and environments by comparing them against predefined ground truth using CLIP similarity. This helps prevent regressions and maintains the quality of diffusion model generations, providing clear feedback when outputs deviate from expected results.

Highlights

New Consistency Testing Framework: Introduced a comprehensive framework for consistency testing of diffusion model outputs in CI, leveraging CLIP similarity to compare generated images/videos against ground truth embeddings.
Ground Truth Management: Implemented logic to load ground truth (GT) data from either a local directory (via SGLANG_CONSISTENCY_GT_DIR environment variable) or a remote sgl-test-files repository, supporting both image and video modalities.
Diffusers Version Check: Added a mechanism to warn if the currently installed diffusers library version does not match the version used to generate the ground truth, helping to identify potential inconsistencies due to library updates.
Configurable CLIP Thresholds: Provided a consistency_threshold.json file to define default CLIP similarity thresholds for image and video outputs, with support for per-test-case overrides.
Integration into Diffusion Tests: Integrated the new consistency validation step into the existing test_diffusion_generation flow, ensuring that all diffusion model outputs are checked for consistency during CI.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

python/sglang/multimodal_gen/test/server/consistency_threshold.json
- Added a new JSON configuration file to store diffusers_version and default CLIP similarity thresholds for image and video consistency checks.
python/sglang/multimodal_gen/test/server/test_server_common.py
- Imported new utility functions for consistency testing.
- Added a new private method _validate_consistency to handle the core logic of comparing generated content with ground truth using CLIP similarity, including environment variable checks, GT loading, and detailed failure reporting.
- Integrated the _validate_consistency method into test_diffusion_generation to perform consistency checks after content generation.
python/sglang/multimodal_gen/test/test_utils.py
- Imported dataclass, requests, and Any for new data structures and network requests.
- Imported DiffusionTestCase for type hinting and test case configuration.
- Added constants for remote ground truth base URL, local threshold JSON path, CLIP model name, and default CLIP thresholds.
- Implemented _load_threshold_json to read consistency configuration.
- Added check_diffusers_version_match to warn about diffusers version discrepancies.
- Created get_clip_threshold to retrieve appropriate similarity thresholds.
- Defined ConsistencyResult dataclass to structure comparison outcomes.
- Developed get_clip_model for lazy-loading and caching the CLIP model.
- Implemented compute_clip_embedding and compute_clip_similarity for core CLIP operations.
- Added get_consistency_gt_candidates to generate potential ground truth filenames.
- Created _get_consistency_gt_dir to check for local ground truth directory configuration.
- Implemented load_gt_embeddings to fetch and process ground truth images/videos into CLIP embeddings.
- Added _remote_gt_url and gt_exists to verify the presence of ground truth files.
- Developed compare_with_gt to perform the actual comparison between generated frames and ground truth embeddings.

Activity

No specific activity (comments, reviews, progress) has been recorded for this pull request yet.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces a robust consistency checking mechanism for diffusion model outputs within the CI pipeline, which is a valuable addition for ensuring output stability. The implementation is well-structured, adding a new configuration file for thresholds, utility functions for ground truth management and comparison using CLIP, and integrating this validation into the existing test suite. The approach to skip tests when ground truth is missing is a good design choice to avoid blocking CI for new test cases. I have a couple of suggestions to enhance the robustness of network requests by using more specific exception handling.

gemini-code-assist · 2026-03-08T19:32:44Z

python/sglang/multimodal_gen/test/test_utils.py

+        try:
+            r = requests.head(url, timeout=10)
+            return url if r.status_code == 200 else None
+        except Exception:
+            return None


The except Exception: is too broad and can hide non-network related errors, making debugging difficult. It's better to catch a more specific exception, like requests.exceptions.RequestException, to only handle network issues.

Suggested change

try:

r = requests.head(url, timeout=10)

return url if r.status_code == 200 else None

except Exception:

return None

try:

r = requests.head(url, timeout=10)

return url if r.status_code == 200 else None

except requests.exceptions.RequestException:

return None

gemini-code-assist · 2026-03-08T19:32:44Z

python/sglang/multimodal_gen/test/test_utils.py

+        try:
+            r = requests.head(url, timeout=10)
+            if r.status_code == 200:
+                return url
+        except Exception:
+            continue


Similar to the above, using except Exception: is too broad. Catching requests.exceptions.RequestException is more specific and safer, as it won't suppress unrelated programming errors.

Suggested change

try:

r = requests.head(url, timeout=10)

if r.status_code == 200:

return url

except Exception:

continue

try:

r = requests.head(url, timeout=10)

if r.status_code == 200:

return url

except requests.exceptions.RequestException:

continue

mickqian

/tag-and-rerun-ci

yhyang201 · 2026-03-09T15:13:03Z

/tag-and-rerun-ci

yhyang201 · 2026-03-09T15:48:30Z

/rerun-failed-ci

yhyang201 · 2026-03-09T16:05:19Z

/rerun-failed-ci

yhyang201 · 2026-03-09T16:28:40Z

/rerun-failed-ci

mickqian · 2026-03-10T02:54:32Z

the amd tests are failing

yhyang201 · 2026-03-16T04:08:56Z

/tag-and-rerun-ci

yhyang201 · 2026-03-17T23:18:27Z

/rerun-failed-ci

yhyang201 · 2026-03-17T23:42:39Z

/rerun-failed-ci

yhyang201 · 2026-03-18T00:04:28Z

/rerun-failed-ci

Prozac614 and others added 2 commits March 8, 2026 19:22

upd

8bc033e

yhyang201 requested review from mickqian and ping1jing2 as code owners March 8, 2026 19:25

github-actions bot added the diffusion SGLang Diffusion label Mar 8, 2026

gemini-code-assist bot reviewed Mar 8, 2026

View reviewed changes

mickqian approved these changes Mar 9, 2026

View reviewed changes

github-actions bot added the run-ci label Mar 9, 2026

yhyang201 changed the title ~~diffusion: pr ci consistency~~ [do not merge] diffusion: pr ci consistency Mar 12, 2026

mickqian mentioned this pull request Mar 12, 2026

[diffusion]: add E2E correctness tests for image and video models #16024

Closed

6 tasks

Merge branch 'main' into pr-ci-consistency

deb4d60

Conversation

yhyang201 commented Mar 8, 2026

Motivation

Modifications

Accuracy Tests

Benchmarking and Profiling

Checklist

Review Process

Uh oh!

gemini-code-assist bot commented Mar 8, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Mar 8, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Mar 8, 2026

Choose a reason for hiding this comment

Uh oh!

mickqian left a comment

Choose a reason for hiding this comment

Uh oh!

yhyang201 commented Mar 9, 2026

Uh oh!

yhyang201 commented Mar 9, 2026

Uh oh!

yhyang201 commented Mar 9, 2026

Uh oh!

yhyang201 commented Mar 9, 2026

Uh oh!

mickqian commented Mar 10, 2026

Uh oh!

yhyang201 commented Mar 16, 2026

Uh oh!

yhyang201 commented Mar 17, 2026

Uh oh!

yhyang201 commented Mar 17, 2026

Uh oh!

yhyang201 commented Mar 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants