Repository navigation
feat(operator): allow EPP replicas in Dynamo mode - #14758
Merged
Merged
Conversation
github-actions
Bot
added
feat
documentation
Contributor
The Gateway API routing reference described operator-generated EPP resources without stating which EPP mode they apply to, and documented spec.components[].replicas as something to increase for more endpoint-picker capacity. Admission rejects any value other than 1, and EPP replication is reachable only in standalone mode, where DYN_EPP_PEER_SERVICE enables peer discovery and cross-replica active-load sync. Readers could therefore take the operator chart's EPP RBAC as the EPP's RBAC in general and bind a standalone EPP to it. That ClusterRole covers only what a Dynamo-mode EPP reads; a standalone deployment ships its own ServiceAccount and Role. Add an EPP Modes section contrasting the two modes and stating that standalone mode runs no operator, correct the replicas field to the value admission enforces, and note that DYN_ROUTER_REPLICA_SYNC has no peer under the single-replica cap. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Anna Tchernych <atchernych@nvidia.com>
Admission rejected any EPP component with replicas other than 1, so the endpoint picker could neither scale for throughput nor survive the loss of its pod. The restriction was not justified by a router limitation: the Frontend hosts the same KV router, is replicated routinely, and carries no such rule. It was the only per-component replica restriction in the validator. The machinery a replicated EPP needs is already in place. Prefix-cache state needs no synchronization, because workers broadcast KV events over the event plane and every subscriber receives all of them, so each replica builds an equivalent radix tree. Active-sequence state is shared by DYN_ROUTER_REPLICA_SYNC, which reaches the EPP's routers through try_kv_router_config_from_dynamo_env and runs the same ActiveSequencesMultiWorker replica sync the Frontend uses. Remove the admission rule and default DYN_ROUTER_REPLICA_SYNC to true when replicas exceeds 1, so a scale-out does not silently leave each replica scoring workers from only the load it placed itself. podTemplate env keeps precedence, so the isolated-view mode the Frontend also offers stays reachable. The legacy Go EPP never receives the setting. Document the accuracy cost in the Gateway API routing reference, including the EPP-specific limit the Frontend does not have: the EPP only selects while Envoy dispatches, so a booking lives for the ext_proc stream rather than the backend request. A retry that lands on another replica frees the original booking while work continues and books again elsewhere, and expiry bounds the error without correcting it. Also adjusts the replication statements added in the preceding commit, which described the single-replica cap this change removes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Anna Tchernych <atchernych@nvidia.com>
atchernych
force-pushed
the
feat/epp-replicas-dynamo-mode
branch
from
September 12, 2026 00:34
c3ae313 to
8d02695
Compare
Signed-off-by: Anna Tchernych <atchernych@nvidia.com>
Signed-off-by: Anna Tchernych <atchernych@nvidia.com>
Contributor
Author
Same-node rerunFollow-up to the perf run above. Today I pinned both deployments' prefill and decode workers to the same B200 node and ran the same sweep again, alternating
On the same node the branch was the slower side this time, but the extra time is again already present at the decode pod's sidecar, before the Gateway and EPP. The Gateway + EPP hop costs about 10 ms on both sides. The remaining difference comes from which GPUs each side's workers landed on and what else runs on that node, not from this PR. |
tmonty12
approved these changes
Sep 23, 2026
Signed-off-by: Anna Tchernych <atchernych@nvidia.com>
atchernych
enabled auto-merge (squash)
September 23, 2026 21:40
harryskim
reviewed
Sep 23, 2026
harryskim
left a comment
harryskim
left a comment
Contributor
There was a problem hiding this comment.
Docs review. No blockers. The doc matches the code (Go EPP detection via eppConfig, sync off by default, ZMQ as the default event plane), and the links and anchors resolve. A few wording and content fixes are inline.
Co-authored-by: Harry Kim <harry_kim@live.com> Signed-off-by: atchernych <atchernych@nvidia.com>
Trim the DYN_ROUTER_REPLICA_SYNC description and list, in the EPP replication section, which router state is shared across replicas, which is best-effort, and which stays per replica. Mechanism details are linked from router-operations.md rather than repeated. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Anna Tchernych <atchernych@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
Targets a later release, not 1.5.0. #14756 documents the single-replica limit as it stands in 1.5.0 and should land on its own; this PR changes that behaviour for a subsequent release.
Summary
The EPP in the standalone mode allows replication but the EPP in the dynamo mode only allowed replica 1. This PR addresses this limitation.
Prior to this PR Admission was rejected any EPP component with
replicasother than 1, so the endpoint picker could neither scale for throughput nor survive the loss of its pod. That restriction was not justified by a router limitation. The Frontend hosts the same KV router and allows for replication:This PR does not introduce any new impresisions. It extends the current frontEnd behavior to the EPP in the dynamo mode.
subscribe, notqueue_subscribe), so each replica builds an equivalent radix tree. Cache-overlap scoring, which dominates worker choice on prefix-heavy traffic, stays as accurate with N replicas as with one.DYN_ROUTER_REPLICA_SYNC, IF ENABLED just like in the frontEnd, which reaches the EPP's decode and prefill routers throughtry_kv_router_config_from_dynamo_envand runs the sameActiveSequencesMultiWorkerreplica sync the Frontend uses.Changes
EPP component must have exactly 1 replicaadmission rule.DYN_ROUTER_REPLICA_SYNCoff by default, matching the Frontend-hosted router.Validation
go test ./...indeploy/operator— 42 packages pass, no failures.replicas: 2to assert acceptance, and the field-path test that pinnedspec.components[].replicasamong the EPP validator's emitted paths.python3 docs/fern/scripts/docs_lint.py --scan docs— 0 errors (8 pre-existing warnings on untouched files).fern check/fern docs broken-linksnot run: the CLI is not installed locally.The magnitude of the selection-to-booking race at realistic arrival rates needs measurement.
Related Issues
🚫 This PR is NOT linked to an issue:
🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Documentation