[P/D Disaggregation] PDController and PDWorker Prototype (1p1d)

[robertgshaw2-redhat]

SUMMARY:

• Implement PDController
  • conforms to EngineClient protocol (thus can be used from OpenAI Server)
  • Send prefill request to Prefill PDWorker, waits
  • Sends decode request to Decode PDWorker, forwards responses to user
• Implement PDWorker - which wraps AsyncLLM to get messages from PDController
• ZMQ based messaging between the two

Design Considerations

NOTE: these are intended as prototype, we may ultimately need to move these implementations into higher level frameworks like DYNAMO or otherwise

Successor to: https://github.com/vllm-project/vllm/pull/11791/files - @panf2333 happy to move this PR under your account or add you as co-author, whatever your prefer

[youkaichao]

instead of having a connector to connect 1P1D, can we have a manager that listens to P/D engine or entrypoint process register? then we can have dynamic XpYd.

[robertgshaw2-redhat]

I think we should do this after https://github.com/vllm-project/vllm/pull/12957/commits

[robertgshaw2-redhat]

One other thing. Another design would be to have It might make sense to have static P and D service that has workers scale out from there. This might work better inside k8s. Need to think more about it and review the design docs in more detail

[phoenixwu0229]

instead of having a connector to connect 1P1D, can we have a manager that listens to P/D engine or entrypoint process register? then we can have dynamic XpYd.

will it be merged?

[youkaichao]

you can set up a side thread to run a busy loop, to receive register / unregister / heartbeat signals from entrypoint (or i would name it api servers) and engines.

[panf2333]

SUMMARY:

* Implement `PDController`
  
  * conforms to `EngineClient` protocol (thus can be used from OpenAI Server)
  * Send prefill request to Prefill `PDWorker`, waits
  * Sends decode request to Decode `PDWorker`, forwards responses to user

* Implement `PDWorker` - which wraps `AsyncLLM` to get messages from `PDController`

* ZMQ based messaging between the two

Design Considerations

NOTE: these are intended as prototype, we may ultimately need to move these implementations into higher level frameworks like DYNAMO or otherwise

Successor to: https://github.com/vllm-project/vllm/pull/11791/files - @panf2333 happy to move this PR under your account or add you as co-author, whatever your prefer

@robertgshaw2-redhat Thanks Robert, it'd be great if you can move this PR to our account. Let me know if you don't have the write permission.

[github-actions]

This pull request has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this pull request should remain open. Thank you!

[puppetm4st3r]

🚀

[pplmx]

Any progress?