[RyanYun09]

Contributor intent (required human answers)

1. What outcome do you want to achieve?

Make vllm-omni's Omni-DuplexEval content-judge prompt match the official evaluator's GT reference handling, so an absent/empty or duplicated answer1/answer2 cannot produce a malformed or duplicated Reference annotations block.

2. What is the highest-leverage variable for that outcome?

The caller-side reference set passed to build_content_prompt. The prompt builder already matches the pinned official wording; the divergence is that evaluate_sample passes [answer1, answer2] verbatim, while the official pipeline applies non_empty_strings and first-occurrence deduplication before prompt construction.

3. How will this work move that variable?

Add _gt_references in omni_duplex_eval_eval.py to drop None/blank values (strip-based, order-preserving) and deduplicate while keeping the first occurrence, then pass its result at the content-prompt call site. A dedicated regression test captures the judge prompt for answer2 == "", answer1 == answer2, and whitespace-padded references.

4. Whose customer or user needs does this address, and what are those needs?

Users of vllm bench omni-duplex-eval --omni evaluate and anyone comparing vllm-omni scores against the official evaluator or another implementation. They need the same judge-prompt semantics so score comparisons are not polluted by gratuitous empty numbered slots or duplicate references.

Purpose

Before this change, evaluate_sample passed [sample.answer1, sample.answer2] directly into build_content_prompt. With answer2 == "", the judge saw an empty numbered slot (1. ); with answer1 == answer2, it saw the same GT twice. The official OpenBMB/Omni-DuplexEval pipeline filters non-empty strings and deduplicates references before building the same pinned content prompt.

After this change, the content prompt receives only stripped, non-empty, first-occurrence-unique references. The prompt wording and PROTOCOL_PIN are unchanged, and the temporal and proactive-reminder paths are untouched.

The affected code is under the draft benchmarking module contract. This change does not alter public CLI options, defaults, configuration, or supported behavior, so no user-guide update is needed.

Fixes #9054

Test Plan

vLLM Version: 0.32.0

vLLM-Omni Commit: 59a60dc (base 324cb5d)

Environment: OrbStack container vllm-omni-029-up; Python 3.11.16, torch 2.13.0+cu130, pytest 9.1.1, PyAV 18.1.0.

# Regression cases introduced by this PR
PYTHONPATH=/workspace/repo python -m pytest \
  tests/benchmarks/duplex/test_omni_duplex_eval_gt_references.py -q

# Duplex eval + media suites
PYTHONPATH=/workspace/repo python -m pytest \
  tests/benchmarks/duplex/test_omni_duplex_eval_gt_references.py \
  tests/benchmarks/duplex/test_omni_duplex_eval.py \
  tests/benchmarks/duplex/test_omni_duplex_eval_media.py -q

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	⚠️ Failed 2026-10-18T04:30:28.073353Z	7fc0579	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[vllm-omni-review-bot]

This PR appears to belong to: docs/design/module/benchmarking.md.

Module owners: @alex-jw-brooks @Bounty-hunter @NickCao

Routing: @alex-jw-brooks via module of the changed files, CODEOWNERS; @Bounty-hunter via module of the changed files, CODEOWNERS; @NickCao via CODEOWNERS

@RyanYun09, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[vllm-omni-review-bot]

Omni ReviewBot triage note

Automated triage of commit f32760a78cd7 produced:

• Priority: high. Prompt maintainer attention is suggested.

These are automated triage suggestions only — the final decision belongs to the maintainers.

[linyueqian]

Thanks, this matches the official evaluator. In OpenBMB/Omni-DuplexEval, batch_real_time_description.py passes non_empty_strings([answer1, answer2]) (strip, drop empty) and load_reference_texts deduplicates in first-occurrence order, matching what _gt_references does before build_content_prompt. The prompt wording and PROTOCOL_PIN are unchanged.

Validation: on main at b1f11f1 merged with a6affc4, CPU only, test_omni_duplex_eval_gt_references.py plus test_omni_duplex_eval.py gave 31 passed, 5 skipped. The three new cases fail on unpatched omni_duplex_eval_eval.py and pass with the fix. Small note for the description: the test plan runs -k official_gt_reference against test_omni_duplex_eval.py, but the new cases live in test_omni_duplex_eval_gt_references.py.

[natureofnature]

@natureofnature

[natureofnature]

Reviewed the full two-file change and the dataset → content-prompt path against OpenBMB/Omni-DuplexEval at the pinned revision. Empty/whitespace-only references are dropped, duplicates keep their first occurrence, and temporal/reminder prompts remain unchanged. The three prompt-capture regressions cover the reported cases; @linyueqian's 31-pass/5-skip CPU report uses the same patch and unchanged relevant evaluator, dataset and prompt code. Current head checks are green. This round was static review; I did not rerun tests.

【Conclusion: Found no blocking issues in commit 59a60dc.】
【Action recommended: None.】

[Gaohan123]

LGTM. Thanks