Skip to content
Merged
Show file tree
Hide file tree
Changes from 18 commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
49e99f8
docs: clarify EPP mode boundary and single-replica Dynamo mode
atchernych Sep 11, 2026
8d02695
feat(operator): allow EPP replicas in Dynamo mode
atchernych Sep 11, 2026
4dea300
fix(operator): keep the single-replica rule for the legacy Go EPP
atchernych Sep 12, 2026
4156c9e
docs: separate backend load from booking state in the EPP replication…
atchernych Sep 12, 2026
2e9cbf7
perf(epp): drop per-request allocations from the pick path
atchernych Sep 12, 2026
eceb2db
docs: state that EPP replica sync mitigates divergence rather than re…
atchernych Sep 12, 2026
ddbc169
docs: say who owns the replica-sync transport in each EPP mode
atchernych Sep 14, 2026
0c01bff
docs: record that EPP queue and admission state is not replicated
atchernych Sep 14, 2026
0bde857
docs: correct the EPP booking-ownership section
atchernych Sep 14, 2026
cdf6b75
chore(operator): drop redundant comments and an unreachable EPP test …
atchernych Sep 14, 2026
d34314c
fix(operator): leave EPP replica sync off by default
atchernych Sep 14, 2026
73acfcd
docs: stop asserting a booking-ownership contrast that does not hold
atchernych Sep 15, 2026
acd6c12
Fix doc
atchernych Sep 16, 2026
ee1b81b
Merge remote-tracking branch 'origin/main' into feat/epp-replicas-dyn…
atchernych Sep 22, 2026
9fc2388
Revert "perf(epp): drop per-request allocations from the pick path"
atchernych Sep 23, 2026
41d2e21
Improve comments
atchernych Sep 23, 2026
a988968
Add a comment
atchernych Sep 23, 2026
603d1bf
Rm the cimment in gateway-api-routing.mdx
atchernych Sep 23, 2026
88f94c1
Apply suggestion from @harryskim
atchernych Sep 28, 2026
1b6e19e
docs(epp): explain what replication makes approximate
atchernych Sep 28, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -1099,14 +1099,29 @@ func TestDynamoComponentDeploymentValidator_Validate(t *testing.T) {
},
},
{
name: "v1alpha1 EPP requires one replica",
name: "v1alpha1 native Rust EPP accepts more than one replica",
deployment: alphaDCDWithSharedSpec(nvidiacomv1alpha1.DynamoComponentDeploymentSharedSpec{
ComponentType: consts.ComponentTypeEPP,
Replicas: &validMinAvail,
ExtraPodSpec: &nvidiacomv1alpha1.ExtraPodSpec{
MainContainer: &corev1.Container{Image: frontendImage150},
},
}),
},
{
name: "v1alpha1 legacy Go EPP still requires one replica",
deployment: alphaDCDWithSharedSpec(nvidiacomv1alpha1.DynamoComponentDeploymentSharedSpec{
ComponentType: consts.ComponentTypeEPP,
Replicas: &validMinAvail,
EPPConfig: &nvidiacomv1alpha1.EPPConfig{
ConfigMapRef: &corev1.ConfigMapKeySelector{
LocalObjectReference: corev1.LocalObjectReference{Name: "legacy-epp-config"},
},
},
ExtraPodSpec: &nvidiacomv1alpha1.ExtraPodSpec{
MainContainer: &corev1.Container{Image: legacyEPPImage140},
},
}),
wantWebhookErrs: []string{
"spec.replicas: Invalid value: 2: EPP component must have exactly 1 replica",
},
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ import (
"github.com/ai-dynamo/dynamo/deploy/operator/internal/consts"
"github.com/ai-dynamo/dynamo/deploy/operator/internal/dra"
"github.com/ai-dynamo/dynamo/deploy/operator/internal/dynamo"
"github.com/ai-dynamo/dynamo/deploy/operator/internal/dynamo/epp"
"github.com/ai-dynamo/dynamo/deploy/operator/internal/features"
"github.com/ai-dynamo/dynamo/deploy/operator/internal/provideroverride"
corev1 "k8s.io/api/core/v1"
Expand Down Expand Up @@ -125,7 +126,8 @@ func (v *sharedValidation) validateDynamoComponentDeploymentSharedSpec(
allErrs = append(allErrs, field.Forbidden(fldPath.Child("type"), fmt.Sprintf("cannot deploy EPP component: %v", err)))
}
}
if spec.Replicas != nil && *spec.Replicas != 1 {
// Rust EPP supports load replication while the Go EPP does not.
if epp.IsLegacyGoEPP(spec.EPPConfig) && spec.Replicas != nil && *spec.Replicas != 1 {
allErrs = append(allErrs, field.Invalid(
fldPath.Child("replicas"),
*spec.Replicas,
Expand Down
28 changes: 22 additions & 6 deletions docs/fern/pages/reference/components/gateway-api-routing.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ pinned to 1.4, does not read `DYN_EPP_MODE` and has no standalone mode.
| Dynamo runtime | Required. It uses whichever discovery and event planes the platform is configured for; etcd and NATS are needed only when selected | Not used |
| Worker discovery | Dynamo runtime discovery | Kubernetes pod watch driven by `InferencePool.spec.selector` |
| EPP ServiceAccount and RBAC | Cluster-wide mode: the operator creates the ServiceAccount and binds it to the `dynamo-platform` chart's EPP ClusterRole. Namespace-restricted mode: the chart creates the ServiceAccount, Role, and RoleBinding | Supplied with the deployment manifests |
| EPP replication | Not available; exactly one replica | Available through `DYN_EPP_PEER_SERVICE` |
| EPP replication | Available. When `DYN_ROUTER_REPLICA_SYNC` is set, replicas exchange active-sequence state through the Dynamo event plane, which the runtime addresses and discovers. It is ZMQ-backed by default and NATS when the runtime is configured for it | Available. Replicas exchange active-sequence state over ZMQ sockets the EPP binds itself, discovering peers from the Service named by `DYN_EPP_PEER_SERVICE` |
Comment thread
atchernych marked this conversation as resolved.
Resolved

A standalone EPP is not managed by the Dynamo operator, so the operator generates no resources for
it. It uses neither the EPP ServiceAccount nor the `dynamo-platform` chart's EPP RBAC. Standalone
Expand Down Expand Up @@ -53,9 +53,13 @@ and is configured through `DYN_*` environment variables. It implements the GAIE
</ParamField>

<ParamField path="spec.components[].replicas" type="integer" default="1">
Number of EPP pods. Must be exactly `1`. Admission rejects any other value with the message
`EPP component must have exactly 1 replica`. EPP replication is not available in Dynamo mode. To run
several EPP replicas that share active-load state, use standalone mode.
Number of EPP pods. Increase it for endpoint-picker throughput or to survive the loss of a single
EPP pod. Above `1`, the routing accuracy of the fleet becomes approximate, and replicas do not share
active-load state unless `DYN_ROUTER_REPLICA_SYNC` is set. Read
[EPP Replication and Routing Accuracy](#epp-replication-and-routing-accuracy) before scaling it.

The deprecated Go EPP, selected by the presence of `eppConfig`, does not support replication. It
must have exactly `1` replica, and admission rejects any other value.
</ParamField>

<ParamField path="spec.components[].frontendSidecar" type="string" required={true}>
Expand Down Expand Up @@ -113,8 +117,10 @@ Gateway topology.

<ParamField path="DYN_ROUTER_REPLICA_SYNC" type="boolean" default="false">
Publishes and consumes best-effort active-sequence state across EPP replicas through the Dynamo event
Comment thread
atchernych marked this conversation as resolved.
Resolved
plane. A DGD EPP component runs a single replica, so this setting has no peer to synchronize with
there.
plane. It is off by default, matching the Frontend-hosted router, and is set through `podTemplate`
env. It has no peer to synchronize with at one replica, and the deprecated Go EPP cannot be
replicated at all. Set this field to `true` when enabling multiple EPP replicas. See
[EPP Replication and Routing Accuracy](#epp-replication-and-routing-accuracy).
Comment thread
atchernych marked this conversation as resolved.
Resolved
</ParamField>

<ParamField path="DYN_ROUTER_TRACK_ACTIVE_BLOCKS" type="boolean" default="true">
Expand All @@ -132,6 +138,16 @@ Gateway topology.
For the full shared configuration surface and tuning guidance, see
[Configuration and Tuning](../../developer-guide/knowledge-base/modular-components/router/configuration-and-tuning.md).

## EPP Replication and Routing Accuracy
Comment thread
atchernych marked this conversation as resolved.
Resolved
Comment thread
atchernych marked this conversation as resolved.
Resolved

Set `DYN_ROUTER_REPLICA_SYNC` to `true` when running more than one EPP replica. Without it, each
replica sees only the load it routed itself, and load balancing across workers suffers.

The EPP embeds the same KV router as the Frontend and reads the same `DYN_ROUTER_*` settings, so
replicated EPPs keep prefix-cache and active-load state the same way replicated Frontends do. See
[Router Operations](../../developer-guide/knowledge-base/modular-components/router/router-operations.md)
for how that state behaves across replicas and what `DYN_ROUTER_REPLICA_SYNC` synchronizes.

## Generated Resource Contract

For a DGD named `<dgd-name>`, the operator generates an `InferencePool` named `<dgd-name>-pool` in
Expand Down
Loading