Skip to content

[BUG] relax pirate persona assertion in test_system_prompt_override#33390

Open
pacoxu wants to merge 1 commit intovllm-project:mainfrom
pacoxu:test-harmony
Open

[BUG] relax pirate persona assertion in test_system_prompt_override#33390
pacoxu wants to merge 1 commit intovllm-project:mainfrom
pacoxu:test-harmony

Conversation

@pacoxu
Copy link
Copy Markdown
Contributor

@pacoxu pacoxu commented Jan 30, 2026

Purpose

A test fix for #32918.

See #32918 (comment)


[2026-01-29T06:14:49Z]         # Verify the response reflects the pirate personality
[2026-01-29T06:14:49Z]         output_text = response.output_text.lower()
[2026-01-29T06:14:49Z]         pirate_indicators = ["arrr", "matey", "ahoy", "ye", "sea"]
[2026-01-29T06:14:49Z]         has_pirate_language = any(
[2026-01-29T06:14:49Z]             indicator in output_text for indicator in pirate_indicators
[2026-01-29T06:14:49Z]         )
[2026-01-29T06:14:49Z] >       assert has_pirate_language, (
[2026-01-29T06:14:49Z]             f"Expected pirate language in response, got: {response.output_text}"
[2026-01-29T06:14:49Z]         )
[2026-01-29T06:14:49Z] E       AssertionError: Expected pirate language in response, got: I’m doing well, thanks for asking! How can I help you today?
[2026-01-29T06:14:49Z] E       assert False
[2026-01-29T06:14:49Z]
[2026-01-29T06:14:49Z] entrypoints/openai/responses/test_harmony.py:1268: AssertionError
[2026-01-29T06:14:49Z] =============================== warnings summary ===============================
[2026-01-29T06:14:49Z] <frozen importlib._bootstrap>:488
[2026-01-29T06:14:49Z]   <frozen importlib._bootstrap>:488: DeprecationWarning: builtin type SwigPyPacked has no __module__ attribute
[2026-01-29T06:14:49Z]
[2026-01-29T06:14:49Z] <frozen importlib._bootstrap>:488
[2026-01-29T06:14:49Z]   <frozen importlib._bootstrap>:488: DeprecationWarning: builtin type SwigPyObject has no __module__ attribute
[2026-01-29T06:14:49Z]
[2026-01-29T06:14:49Z] ../../usr/local/lib/python3.12/dist-packages/schemathesis/generation/coverage.py:305
[2026-01-29T06:14:49Z]   /usr/local/lib/python3.12/dist-packages/schemathesis/generation/coverage.py:305: DeprecationWarning: jsonschema.exceptions.RefResolutionError is deprecated as of version 4.18.0. If you wish to catch potential reference resolution errors, directly catch referencing.exceptions.Unresolvable.
[2026-01-29T06:14:49Z]     ref_error: type[Exception] = jsonschema.RefResolutionError,
[2026-01-29T06:14:49Z]
[2026-01-29T06:14:49Z] -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
[2026-01-29T06:14:49Z] =========================== short test summary info ============================
[2026-01-29T06:14:49Z] FAILED entrypoints/openai/responses/test_harmony.py::test_system_prompt_override[openai/gpt-oss-20b] - AssertionError: Expected pirate language in response, got: I’m doing well, thanks for asking! How can I help you today?
[2026-01-29T06:14:49Z] assert False
[2026-01-29T06:14:49Z] !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
[2026-01-29T06:14:49Z] ======= 1 failed, 45 passed, 1 skipped, 3 warnings in 263.45s (0:04:23) ========
[2026-01-29T06:14:52Z] 🚨 Error: The command exited with status 1
[2026-01-29T06:14:52Z] user command error: The plugin docker command hook exited with status 1

Test Plan

Test Result


Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
  • (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

…ride

Signed-off-by: Paco Xu <paco.xu@daocloud.io>
@mergify mergify bot added the bug Something isn't working label Jan 30, 2026
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request addresses a flaky test by relaxing a strict assertion. The test now correctly validates that either the model's output reflects the requested persona or the reasoning behind the output acknowledges it. This is a more robust approach for testing language models, which can be non-deterministic. The implementation is clean, and the improved assertion message will aid in debugging future test failures. The change is sound and I have no concerns.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant