Skip to content

[CI] fix test: use minimum supported layered output count#2350

Merged
gcanlin merged 1 commit into
vllm-project:mainfrom
david6666666:codex/layers-guard-min3
Mar 31, 2026
Merged

[CI] fix test: use minimum supported layered output count#2350
gcanlin merged 1 commit into
vllm-project:mainfrom
david6666666:codex/layers-guard-min3

Conversation

@david6666666
Copy link
Copy Markdown
Collaborator

@david6666666 david6666666 commented Mar 31, 2026

PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTTOM) HAVE BEEN CONSIDERED.

Purpose

fix CI

__________________________________________________ test_layered_output_image_count[layers_guard_001_layers2] ___________________________________________________
omni_server = <tests.conftest.OmniServer object at 0x77460033d400>, openai_client = <tests.conftest.OpenAIClientHandler object at 0x77460033fbc0>
expected_layers = 2
    @pytest.mark.advanced_model
    @pytest.mark.diffusion
    @pytest.mark.parametrize(
        "omni_server, expected_layers",
        LAYERS_GUARD_CASES,
        indirect=["omni_server"],
    )
    def test_layered_output_image_count(
        omni_server: OmniServer,
        openai_client: OpenAIClientHandler,
        expected_layers: int,
    ):
        """Guard for https://github.com/vllm-project/vllm-omni/issues/1969
        The bug: API flattened the multi-layer output into a single image.
        Expected: API response content must be a list containing exactly
        `layers image_url items, one per decomposed layer.
        """
        image_data_url = f"data:image/jpeg;base64,{generate_synthetic_image(512, 512)['base64']}"
        messages = dummy_messages_from_mix_data(image_data_url=image_data_url, content_text=EDIT_PROMPT)
        request_config = {
            "model": omni_server.model,
            "messages": messages,
            "extra_body": {
                "num_inference_steps": 2,
                "negative_prompt": NEGATIVE_PROMPT,
                "true_cfg_scale": 4.0,
                "seed": 42,
                "layers": expected_layers,
            },
        }
>       image_items = _collect_image_url_items(openai_client, request_config)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tests/e2e/online_serving/test_qwen_image_layered_expansion.py:180:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/e2e/online_serving/test_qwen_image_layered_expansion.py:119: in _collect_image_url_items
    chat_completion = openai_client.client.chat.completions.create(
/usr/local/lib/python3.12/dist-packages/openai/_utils/_utils.py:286: in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
/usr/local/lib/python3.12/dist-packages/openai/resources/chat/completions/completions.py:1204: in create
    return self._post(
/usr/local/lib/python3.12/dist-packages/openai/_base_client.py:1297: in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <openai.OpenAI object at 0x77460033ef00>, cast_to = <class 'openai.types.chat.chat_completion.ChatCompletion'>
options = FinalRequestOptions(method='post', url='/chat/completions', params={}, headers=NOT_GIVEN, max_retries=NOT_GIVEN, timeo...erence_steps': 2, 'negative_prompt': 'blurry, low quality, distorted', 'true_cfg_scale': 4.0, 'seed': 42, 'layers': 2})
    def request(
        self,
        cast_to: Type[ResponseT],
        options: FinalRequestOptions,
        *,
        stream: bool = False,
        stream_cls: type[_StreamT] | None = None,
    ) -> ResponseT | _StreamT:
        cast_to = self._maybe_override_cast_to(cast_to, options)
        # create a copy of the options we were given so that if the
        # options are mutated later & we then retry, the retries are
        # given the original options
        input_options = model_copy(options)
        if input_options.idempotency_key is None and input_options.method.lower() != "get":
            # ensure the idempotency key is reused between requests
            input_options.idempotency_key = self._idempotency_key()
        response: httpx.Response | None = None
        max_retries = input_options.get_max_retries(self.max_retries)
        retries_taken = 0
        for retries_taken in range(max_retries + 1):
            options = model_copy(input_options)
            options = self._prepare_options(options)
            remaining_retries = max_retries - retries_taken
            request = self._build_request(options, retries_taken=retries_taken)
            self._prepare_request(request)
            kwargs: HttpxSendArgs = {}
            if self.custom_auth is not None:
                kwargs["auth"] = self.custom_auth
            if options.follow_redirects is not None:
                kwargs["follow_redirects"] = options.follow_redirects
            log.debug("Sending HTTP Request: %s %s", request.method, request.url)
            response = None
            try:
                response = self._client.send(
                    request,
                    stream=stream or self._should_stream_response_body(request=request),
                    **kwargs,
                )
            except httpx.TimeoutException as err:
                log.debug("Encountered httpx.TimeoutException", exc_info=True)
                if remaining_retries > 0:
                    self._sleep_for_retry(
                        retries_taken=retries_taken,
                        max_retries=max_retries,
                        options=input_options,
                        response=None,
                    )
                    continue
                log.debug("Raising timeout error")
                raise APITimeoutError(request=request) from err
            except Exception as err:
                log.debug("Encountered Exception", exc_info=True)
                if remaining_retries > 0:
                    self._sleep_for_retry(
                        retries_taken=retries_taken,
                        max_retries=max_retries,
                        options=input_options,
                        response=None,
                    )
                    continue
                log.debug("Raising connection error")
                raise APIConnectionError(request=request) from err
            log.debug(
                'HTTP Response: %s %s "%i %s" %s',
                request.method,
                request.url,
                response.status_code,
                response.reason_phrase,
                response.headers,
            )
            log.debug("request_id: %s", response.headers.get("x-request-id"))
            try:
                response.raise_for_status()
            except httpx.HTTPStatusError as err:  # thrown on 4xx and 5xx status code
                log.debug("Encountered httpx.HTTPStatusError", exc_info=True)
                if remaining_retries > 0 and self._should_retry(err.response):
                    err.response.close()
                    self._sleep_for_retry(
                        retries_taken=retries_taken,
                        max_retries=max_retries,
                        options=input_options,
                        response=response,
                    )
                    continue
                # If the response is streamed then we need to explicitly read the response
                # to completion before attempting to access the response text.
                if not err.response.is_closed:
                    err.response.read()
                log.debug("Re-raising status error")
>               raise self._make_status_error_from_response(err.response) from None
E               openai.BadRequestError: Error code: 400 - {'error': {'message': 'Invalid layers value 2. layers must be between 3 and 10 inclusive.', 'type': 'BadRequestError', 'param': None, 'code': 400}}
/usr/local/lib/python3.12/dist-packages/openai/_base_client.py:1070: BadRequestError
======================================================================= warnings summary =======================================================================
<frozen importlib._bootstrap>:488
  <frozen importlib._bootstrap>:488: DeprecationWarning: builtin type SwigPyPacked has no __module__ attribute
<frozen importlib._bootstrap>:488
  <frozen importlib._bootstrap>:488: DeprecationWarning: builtin type SwigPyObject has no __module__ attribute
../../../usr/local/lib/python3.12/dist-packages/torch/jit/_script.py:362: 14 warnings
  /usr/local/lib/python3.12/dist-packages/torch/jit/_script.py:362: DeprecationWarning: torch.jit.script_method is deprecated. Please switch to torch.compile or torch.export.
    warnings.warn(
vllm_omni/__init__.py:29
  /workspace/build/buildkite/vllm_omni/__init__.py:29: RuntimeWarning: Failed to import version from _version.py: No module named 'vllm_omni._version'
  This typically happens in development mode before building.
  Using fallback version 'dev'.
    from .version import __version__, __version_tuple__  # isort:skip
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=================================================================== short test summary info ====================================================================
FAILED tests/e2e/online_serving/test_qwen_image_layered_expansion.py::test_layered_output_image_count[layers_guard_001_layers2] - openai.BadRequestError: Error code: 400 - {'error': {'message': 'Invalid layers value 2. layers must be between 3 and 10 inclusive.', 'type': 'BadRequestError', 'param': None, 'code': 400}}
============================================ 1 failed, 38 passed, 61 deselected, 17 warnings in 5765.57s (1:36:05) =============================================

Test Plan

pytest tests/e2e/online_serving/test_qwen_image_layered_expansion.py -v

Test Result

======================================================= test session starts =======================================================
platform linux -- Python 3.12.3, pytest-9.0.2, pluggy-1.6.0 -- /workspace/c00580271/cwq_branch/.venv/bin/python3
cachedir: .pytest_cache
rootdir: /workspace/c00580271/cwq_branch/vllm-omni
configfile: pyproject.toml
plugins: anyio-4.12.1
collected 4 items                                                                                                                 

tests/e2e/online_serving/test_qwen_image_layered_expansion.py::test_feature[sp_001] PASSED                                  [ 25%]
tests/e2e/online_serving/test_qwen_image_layered_expansion.py::test_empty_prompt[prompt_001] PASSED                         [ 50%]
tests/e2e/online_serving/test_qwen_image_layered_expansion.py::test_feature[cfg_parallel_001] PASSED                        [ 75%]
tests/e2e/online_serving/test_qwen_image_layered_expansion.py::test_layered_output_image_count[layers_guard_001_layers3] PASSED [100%]


Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan. Please provide the test scripts & test commands. Please state the reasons if your codes don't require additional test scripts. For test file guidelines, please check the test style doc
  • The test results. Please paste the results comparison before and after, or the e2e results.
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model. Please run mkdocs serve to sync the documentation editions to ./docs.
  • (Optional) Release notes update. If your change is user-facing, please update the release notes draft.

BEFORE SUBMITTING, PLEASE READ https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md (anything written below this line will be removed by GitHub Actions)

Signed-off-by: David Chen <530634352@qq.com>
@gcanlin
Copy link
Copy Markdown
Collaborator

gcanlin commented Mar 31, 2026

Copy link
Copy Markdown
Collaborator

@gcanlin gcanlin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@gcanlin gcanlin merged commit 0cbd1cb into vllm-project:main Mar 31, 2026
7 of 8 checks passed
vraiti pushed a commit to vraiti/vllm-omni that referenced this pull request Apr 9, 2026
lengrongfu pushed a commit to lengrongfu/vllm-omni that referenced this pull request May 1, 2026
clodaghwalsh17 pushed a commit to clodaghwalsh17/nm-vllm-omni-ent that referenced this pull request May 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready label to trigger buildkite CI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants