[Core] Encoder separation for Encode-Prefill-Decode Disaggregation

[fake0fan]

Disaggregated Encoder

A disaggregated encoder runs the vision-encoder stage of a multimodal LLM in a process that is separate from the prefill / decoder stage. Deploying these two stages in independent vLLM instances brings three practical benefits:

1. Independent, fine-grained scaling
2. Lower time-to-first-token (TTFT)
3. Cross-process reuse and caching of encoder outputs

We received a series of feedback from the 21740 (Many thanks to @LastZhabka for the excellent work on the initial version), and based on this feedback, we updated our overall design logic.

Design doc: https://docs.google.com/document/d/1aed8KtC6XkXtdoV87pWT0a8OJlZ-CpnuLLzmR8l9BAE

@ywang96 @NickLucche @DarkLight1337 @WoosukKwon

CLOSE #20799

1 New Encoder Cache Connector

Disaggregated encoding in v1 is enabled by an explicit EC connector abstraction. The ECConnectorBase defines a clear split of responsibilities between the scheduler-side and worker-side, and provides the lifecycle hooks required to check, load, save, and track encoder cache (EC) across processes.

ECConnector – interface for retrieving EC caches produced by the encoder.

• Scheduler role – checks cache existence and schedules loads. (lives in the scheduler process):

  • check_caches_exist(request) -> list[bool]: probe whether EC exists for each multimodal datum in the request.
  • update_state_after_alloc(request, index): record EC loading intent once encoder cache space is allocated for a given mm datum.
  • build_connector_meta(scheduler_output) -> ECConnectorMetadata: build per-step metadata consumed by workers; also resets connector internal state for the next step.
  • request_finished(request) -> (bool, Optional[dict]): optionally indicate async save/send in progress and attach metadata.
  • update_connector_output(connector_output): ingest worker-side completion info.

• Worker role – loads the embeddings into memory. (lives in each worker process):

  • bind_connector_metadata(meta) / clear_connector_metadata(): attach per-step metadata from the scheduler before/after forward.
  • start_load_caches(**kwargs): load required EC into the local encoder cache before embeddings are gathered.
  • save_caches(**kwargs): persist/transfer EC produced on the encoder.
  • wait_for_save(): block until outstanding saves complete (if any).
  • get_finished(finished_req_ids) -> (set[str]|None, set[str]|None): report completion of async send/recv to help scheduler bookkeeping.

---

2 Change highlights (Scheduler and GPUModelRunner)

• Scheduler (vllm/v1/core/sched/scheduler.py):

  • Initializes an EC connector when ec_transfer_config is present via ECConnectorFactory.create_connector(..., role=SCHEDULER).
  • Tracks an encoder-token budget and skips EC work when cache already exists (local or remote as reported by the connector).
  • After allocation, calls ec_connector.update_state_after_alloc(...) so the next build_connector_meta(...) includes the mm hashes to load.
  • Emits ec_connector_metadata in SchedulerOutput for the workers, and ingests worker completion via ec_connector.update_connector_output(...).

• Worker (vllm/v1/worker/gpu_model_runner.py):

  • Encoder-side (producer):
    • Within execute_model, when get_ec_transfer().is_producer is True, the runner enters with maybe_get_ec_connector_output(..., encoder_cache=self.encoder_cache): before running the multimodal encoder.
    • The encode pass computes embeddings and writes them into encoder_cache[mm_hash].
    • Immediately after finishing the encode for a given mm_hash, the runner calls maybe_save_ec_to_connector(self.encoder_cache, mm_hash) which invokes ECConnectorBase.save_caches(encoder_cache=..., mm_hash=...).
    • On context exit, wait_for_save() is invoked (if enabled) to ensure the persisted EC is durable/visible to consumers; get_finished(...) is queried to surface completion status back to the scheduler.
  • PD-side (consumer):
    • For requests scheduled on PD, the scheduler supplies ec_connector_metadata that lists the mm_hash items needing loads.
    • The runner binds this metadata and calls start_load_caches(encoder_cache=self.encoder_cache) prior to _gather_mm_embeddings, allowing the connector to populate encoder_cache[mm_hash] from the external store.
    • _gather_mm_embeddings then reads the loaded tensors from encoder_cache and returns them as multimodal embeddings for the subsequent decoder input embedding construction.
    • After the forward step, the runner clears metadata; any connector-furnished completion info is recorded into ECConnectorOutput for the scheduler to free resources when safe.

---

3 Usage Example

The current reference pathway is ECSharedStorageConnector.
Below ready-to-run scripts show the workflows:

• 1 Encoder + 1 PD:

  • examples/online_serving/disaggregated_encoder/shared_storage_connector/disagg_1e1pd_example.sh
  • (legacy/simple demo) examples/online_serving/disaggregated_encoder/shared_storage_connector/disagg_encoder_example.sh

• 1 Encoder + 1 Prefill + 1 Decode:

  • examples/online_serving/disaggregated_encoder/shared_storage_connector/disagg_1e1p1d_example.sh

3.1 Minimal ECTransfer CLI config

The Encoder and PD share/transfer encoder cache (EC) via EC transfer. The current reference implementation is ECSharedStorageConnector (dump/load via a shared directory).

• Encoder (producer) example:

vllm serve <model> \
  --ec-transfer-config '{
    "ec_connector": "ECSharedStorageConnector",
    "ec_role": "ec_producer",
    "ec_connector_extra_config": {"shared_storage_path": "/tmp"}
  }'

• PD (consumer) example:

vllm serve <model> \
  --ec-transfer-config '{
    "ec_connector": "ECSharedStorageConnector",
    "ec_role": "ec_consumer",
    "ec_connector_extra_config": {"shared_storage_path": "/tmp"}
  }'

Notes:

• If shared_storage_path is not explicitly set, /tmp is used by default.
• This connector persists EC per mm_hash to /<shared_storage_path>/<mm_hash>/encoder_cache.safetensors, and the PD side loads it to GPU on demand.
• Connector selection and loading are handled by ECConnectorFactory (with ECSharedStorageConnector registered by default).

---

4 Development

Here is a figure illustrating disaggregate encoder flow:

Disaggregated encoding is implemented by running two parts:

• Encoder instance – a vLLM instance to performs vision encoding.
• Prefill/Decode (PD) instance(s) – runs language pre-fill and decode.
  • PD can be a single instance (E->PD, see disagg_1e1pd_example.sh / disagg_encoder_example.sh), or disaggregated with Decode (E->P->D, see disagg_1e1p1d_example.sh).

A connector transfers encoder-cache (EC) embeddings from the encoder instance to the PD instance.
All related code is under vllm/distributed/ec_transfer.

For the PD disaggregation part, the Prefill instance receives cache exactly the same as the disaggregate encoder flow above. Prefill instance executes 1 step (prefill -> 1 token output) and then transfers KV cache to the Decode instance for the remaining execution. The KV transfer part purely happens after the execution of the Prefill instance. docs/features/disagg_prefill.md shows the brief idea about the disaggregated prefill (v0). We create the example setup with the NixlConnector from vllm/distributed/kv_transfer/kv_connector/v1/nixl_connector.py in vllm V1 and referred to the tests/v1/kv_connector/nixl_integration/toy_proxy_server.py to facilitate the kv transfer between P and D;

5. Next-Step Plan

5.1 Broaden EC Connector Types

5.2 Multi-Hardware Platform Support

5.3 Comprehensive Performance Evaluation

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @fake0fan.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[ywang96]

Per offline discussion, since we're not going to include this in the 0.11.1 release, let's update this PR and get it in if all CI tests pass!

Very much appreciate everyone's efforts on pushing this PR to the finishing line 🚀 🚀 🚀

[gty111]

We really appreciate the great work in this PR. We’d really like to have it merged into the main branch.

[chaunceyjiang]

Nice! @fake0fan

[npuichigo]

Do we have any plan to refine the Whisper implementation on benefit from EPD disaggregation?