Skip to content

fix(dcp): build the indexer query-split topology for a single shard - #244

Closed
koush wants to merge 1 commit into
local-inference-lab:dev/gilded-gnosisfrom
koush:fix/dcp-indexer-shards1-query-split
Closed

fix(dcp): build the indexer query-split topology for a single shard#244
koush wants to merge 1 commit into
local-inference-lab:dev/gilded-gnosisfrom
koush:fix/dcp-indexer-shards1-query-split

Conversation

@koush

@koush koush commented Aug 6, 2026

Copy link
Copy Markdown

VLLM_DCP_INDEXER_SHARDS=1 replicates the sparse-indexer K cache on every DCP rank, but the indexer topology was only built for 1 < shards < DCP, so one-shard runs fell back to _QUERY_SPLIT, which is singleton at TP == DCP. Every rank then scored every query row against the full indexer cache during prefill: DCP-wide redundant compute (~8x at DCP8, measurable as a long-context prefill throughput loss). With VLLM_DCP_QUERY_SPLIT=1 it was worse than redundant, since get_indexer_query_split_group(1) had no group to return and raised "No indexer query-split group matches the requested KV shard count".

_build_indexer_replica_group_ranks already yields the correct layout for shards=1 -- single-rank owner groups and one TP-wide query-split group -- so let it run. Each rank scores totalQ/TP rows against the full replicated cache, and the existing indices-only all-gather restores the shared top-k buffer. No merge is needed or attempted: dcp_world_size stays 1 for a replicated cache, so every candidate-merge path still early-returns.

Claude-Session: https://claude.ai/code/session_01J4BQPVrK6LdTUbXudyPkga

PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTTOM) HAVE BEEN CONSIDERED.

Purpose

Test Plan

Test Result


Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

BEFORE SUBMITTING, PLEASE READ https://docs.vllm.ai/en/latest/contributing (anything written below this line will be removed by GitHub Actions)

Summary by CodeRabbit

  • Bug Fixes
    • Improved distributed processing initialization for configurations using a single indexer shard with multiple decode-context parallel partitions.
    • Ensures these configurations now initialize correctly and operate as expected.

VLLM_DCP_INDEXER_SHARDS=1 replicates the sparse-indexer K cache on every
DCP rank, but the indexer topology was only built for
1 < shards < DCP, so one-shard runs fell back to _QUERY_SPLIT, which is
singleton at TP == DCP. Every rank then scored every query row against
the full indexer cache during prefill: DCP-wide redundant compute (~8x
at DCP8, measurable as a long-context prefill throughput loss). With
VLLM_DCP_QUERY_SPLIT=1 it was worse than redundant, since
get_indexer_query_split_group(1) had no group to return and raised
"No indexer query-split group matches the requested KV shard count".

_build_indexer_replica_group_ranks already yields the correct layout for
shards=1 -- single-rank owner groups and one TP-wide query-split group --
so let it run. Each rank scores totalQ/TP rows against the full
replicated cache, and the existing indices-only all-gather restores the
shared top-k buffer. No merge is needed or attempted: dcp_world_size
stays 1 for a replicated cache, so every candidate-merge path still
early-returns.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J4BQPVrK6LdTUbXudyPkga
@coderabbitai

coderabbitai Bot commented Aug 6, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 58659ecb-a678-413b-8999-fa6fb5b488c9

📥 Commits

Reviewing files that changed from the base of the PR and between 3003860 and 52cece6.

📒 Files selected for processing (1)
  • vllm/distributed/parallel_state.py

📝 Walkthrough

Walkthrough

The indexer replica-group initialization condition now supports indexer_shards == 1 when the decode-context parallel size is greater than one.

Changes

Indexer DCP initialization

Layer / File(s) Summary
Expand indexer replica-group condition
vllm/distributed/parallel_state.py
The setup condition includes single-shard indexer configurations with multi-way DCP. Zero and full DCP shard counts remain excluded.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Possibly related PRs

Suggested reviewers: voipmonitor, lukealonso, yewentao256

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the fix for building DCP indexer query-split topology with a single shard.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@voipmonitor

Copy link
Copy Markdown

Superseded by #245. The original functional commit is preserved unchanged; the replacement adds focused regression coverage for the indexer_shards=1 initialization boundary and documents the completed TP4/DCP4 topology and A-B-A E2E validation. A separate PR was required because GitHub rejected a direct maintainer push to this fork branch with HTTP 403 despite maintainer_can_modify=true.

@voipmonitor voipmonitor closed this Aug 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants