[gty111]

Summary

Batch proxy-to-encoder HTTP requests: instead of sending one request per image, the proxy groups images from the same user request assigned to the same E instance into a single multi-image request. Per-image round-robin assignment and parallel requests to different E instances are preserved. Audio/video remain unbatched; no new configuration is added.

Return item positions in EC metadata so the proxy can map each result back to the original image, including repeated images and processing-derived hashes. This changes request fan-out in the proxy, not encoder compute batching; no scheduler or model-runner changes.

Results

Same-node A/B: Qwen3.5-35B-A3B BF16, 4E+4PD on two GB200 nodes, Mooncake RDMA, MRv2, concurrency 96, CUDA graphs up to 16384 tokens, no eager. Each arm: 200 warmup requests, then 2 × 1200 MUIRBench requests with structured choice.

Metric	Baseline	Batched	Change
Mean throughput	90.22 req/s	96.34 req/s	+6.8%
Mean latency	1035.7 ms	975.2 ms	-5.8%
P99 latency	3036.0 ms	2862.8 ms	-5.7%
Accuracy	58.92%	59.33%	+0.42 pp

E subrequests per round: 4924 → 3878 (-21.2%). All requests succeeded without proxy retries. No accuracy regression observed; this is a two-round comparison, not a claim of improved model accuracy. Both arms use the same worker metadata addition and differ only in the proxy.

Validation

.venv/bin/python -m pytest tests/v1/ec_connector/unit/{test_epd_proxy_retry,test_epd_proxy_round_robin,test_ec_example_connector}.py -q

58 passed; pre-commit passed. Covers repeated images, derived hashes, raw-media mode, metadata ordering and HTTP retries.

Checked #52409 and open PRs for encoder batching/fan-out; no duplicate found. #54176 concerns dynamic registration, not request batching.

AI assistance was used. Draft pending human review.

[mergify]

Documentation preview: https://vllm--57095.org.readthedocs.build/en/57095/

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @gty111.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[jiaran-king]

LGTM. I went through the PR again after the latest fix and don’t see any remaining blockers.

[gty111]

/ci run

[github-actions]

✅ Triggered Buildkite CI #89528 for commit 746aa43e2b55.

[mergify]

Hi @gty111, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

[gty111]

/ci run

[github-actions]

✅ Triggered Buildkite CI #89547 for commit 59f4645657f7.

[gty111]

/ci run

[github-actions]

✅ Triggered Buildkite CI #89568 for commit b237d47c58b7.