[Bugfix] fix DP-aware routing in OpenAI API requests

[inkcherry]

Purpose

fix #24945
In add_request, duplicate initialization is skipped, but during the previousself.processor.process_inputs, data_parallel_rank is not initialized. Using -H 'X-data-parallel-rank' to specify the data parallel rank would be invalid in this case., cc @njhill

vllm/vllm/v1/engine/async_llm.py

Lines 283 to 302 in d69062c

	# Convert Input --> Request.
	if isinstance(prompt, EngineCoreRequest):
	request = prompt
	else:
	assert prompt_text is None
	logger.warning_once(
	"Processor has been moved under OpenAIServing and will "
	"be removed from AsyncLLM in v0.13."
	)
	request = self.processor.process_inputs(
	request_id,
	prompt,
	params,
	arrival_time,
	lora_request,
	tokenization_kwargs,
	trace_headers,
	priority,
	data_parallel_rank,
	)

Test Plan

Test Result

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[gemini-code-assist]

Code Review

This pull request aims to fix an issue with Data Parallelism-aware routing by propagating the data_parallel_rank to the engine's processor. The changes correctly pass the rank through serving_completion.py and serving_engine.py.

However, I've identified a critical issue in serving_engine.py where the signature of _process_inputs is changed in a way that breaks other parts of the code and has an incorrect type hint. I've left a comment with a suggested fix to make the new argument optional, which will prevent runtime errors.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[markmc]

Nice catch!

/cc @Prowindy

[markmc]

Looks like a similar fix is required in serving_chat.py?

Also would be good to catch this issue in the test_serving_chat_data_parallel_rank_extraction test

Even better, it would be great to add a similar test for serving_completion !

[inkcherry]

@markmc Thanks for the comment. I've added it to serving_chat.py.

I noticed that mock objects won't trigger this error. So I added a test with a real engine for coverage, placed after the test_dp_rank_argument test.

[ChuanLi1101]

Could this be a breaking change for the current API?

[inkcherry]

Actually not. When unspecified, it defaults to None and uses the default DP load algorithm

[Prowindy]

@inkcherry could you clarify what failure you saw, crash or request failure? Any repro steps will be helpful.

I think data_parallel_rank isn't a must-have for all endpoints. Only the endpoints supported by https://github.com/vllm-project/router would have X-http-header set and data_parallel_rank needed. Missing this should fallback to normal routing mode.

[inkcherry]

@inkcherry could you clarify what failure you saw, crash or request failure? Any repro steps will be helpful.

I think data_parallel_rank isn't a must-have for all endpoints. Only the endpoints supported by https://github.com/vllm-project/router would have X-http-header set and data_parallel_rank needed. Missing this should fallback to normal routing mode.

Thank you for your great work and feedback. I noticed that it does not crash; instead, it fails under certain circumstances (equivalent to not being specified). I have added tests to cover this.

I agree with your observation, I did not add support for new endpoints.

[njhill]

Thanks @inkcherry!

Doesn't need to be addressed in this PR but I don't see why we would only want to support this header on the chat and completion endpoints, it could apply similarly to all of the endpoints.

Review comments addressed.