[router] refactor worker to builder pattern 1/n#10628
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Summary of Changes
Hello @slin1237, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request initiates a refactoring effort to introduce the builder pattern for creating worker instances within the router. By abstracting the construction logic into BasicWorkerBuilder and DPAwareWorkerBuilder, it aims to improve the maintainability, readability, and flexibility of worker object instantiation, making it easier to configure workers with various parameters in a fluent manner. This is the first step in a series of changes to enhance the worker management system.
Highlights
- New Worker Builder Module: A new module,
sgl-router/src/core/worker_builder.rs, has been introduced to encapsulate the new builder implementations for worker objects. - BasicWorkerBuilder Implementation: A fluent builder,
BasicWorkerBuilder, is now available for constructingBasicWorkerinstances, allowing for easier configuration of worker type, connection mode, labels, health settings, circuit breaker configurations, and gRPC clients. - DPAwareWorkerBuilder Implementation: A
DPAwareWorkerBuilderhas been added to facilitate the creation ofDPAwareWorkerobjects. This builder internally utilizes theBasicWorkerBuilderto configure the underlying base worker. - DPAwareWorker Constructor Update: A new public constructor,
DPAwareWorker::with_base_worker, was added toDPAwareWorkerto enable its creation with a pre-configuredBasicWorker, specifically to support the new builder pattern. - Comprehensive Unit Tests: Extensive unit tests have been included for both
BasicWorkerBuilderandDPAwareWorkerBuilderto ensure the correct functionality and proper configuration of workers created using these new builders.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in pull request comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
There was a problem hiding this comment.
Code Review
This pull request introduces a builder pattern for creating BasicWorker and DPAwareWorker instances, which is a great step towards improving the ergonomics of worker creation. The new worker_builder module is well-structured with BasicWorkerBuilder and DPAwareWorkerBuilder, and includes a good set of unit tests. The refactoring is clean and the use of a fluent API will make the code more readable. I've found one issue with the DPAwareWorkerBuilder that prevents it from being used with gRPC workers, which I've detailed in a specific comment.
slin1237
force-pushed
the
worker-builder-0
branch
2 times, most recently
from
e0eb559 to
a9a52b8
Compare
slin1237
force-pushed
the
worker-builder-0
branch
from
a9a52b8 to
b89a0ab
Compare
* origin/qwen3: (30 commits) chore: bump sgl-kernel 0.3.11 (sgl-project#10630) feat: add fused moe config for Qwen3-Next-80B-A3B-Instruct on B200 (sgl-project#10631) model support: Sarashina2VisionForCausalLM (sgl-project#10632) [Performance] Qwen3-Next: speed up update_mamba_state_after_mtp_verify by 10x; e2e up to 3.54% faster (sgl-project#10586) [Performance] Qwen3-Next: replace arange to cached query_start_loc_li… (sgl-project#10553) [Feature] Speculative decoding support lookahead (sgl-project#9873) refactor: use registry for _get_attention_backend_from_str (sgl-project#10629) [router] refactor worker to builder pattern 1/n (sgl-project#10628) Garbage collector regression in the online server (sgl-project#10621) feat: Add FlexAttention Backend for Efficient Sparse Attention (sgl-project#9947) Fix bias handling in TritonMoeQuantInfo within quantization/mxfp4.py (sgl-project#10579) [Performance] qwen3-next improve causal conv1d in prefill phase (sgl-project#10595) Fix sgl_kernel import failure on devices other than CUDA (sgl-project#10610) support qwen3-next-fp8 deepep (sgl-project#10622) update deepep version for qwen3-next deepep moe (sgl-project#10624) Feat/add heartbeat mechanism for nixl conn (sgl-project#10222) [RL] Add destroy process group api (sgl-project#9979) fix deepep assert when PD disaggregation == null (sgl-project#8274) Scale kkt after reduction (sgl-project#10604) [improvement] add average input/output token length for hicache benchmark stats output (sgl-project#10525) ...
2 participants
Motivation
#10627
Checklist