feat: add priority to backendRef by cmaddalozzo · Pull Request #673 · envoyproxy/ai-gateway

cmaddalozzo · 2025-06-03T14:03:39Z

Commit Message

Add priority field to AIGatewayRouteRuleBackendRef. This will set priority on the underlying LocalityLB endpoint in Envoy.

Related Issues/PRs (if applicable)

N/A

Special notes for reviewers (if applicable)

In order to get the Envoy Gateway to reconcile the underlying HTTPRoute when the priorities in the AIGatewayRoute spec change we need to add an annotation. This annotation just store the priorities for each rule. This ensure we only trigger reconciliation when the priorities actually change.

api/v1alpha1/api.go

internal/controller/ai_gateway_route.go

mathetake

maybe you might want to update the provider_fallback e2e ?

internal/extensionserver/extensionserver.go

mathetake · 2025-06-04T17:55:18Z

i am glad that the core change is much smaller than i thought! I think the rest is documentation + e2e tests

api/v1alpha1/api.go

cmaddalozzo · 2025-06-04T20:01:36Z

maybe you might want to update the provider_fallback e2e ?

I don't think we will be able to add this until the previous priority retry predicate PR makes it into Envoy Gateway.

envoyproxy/gateway#6204

mathetake · 2025-06-04T20:04:17Z

hmm i was under the impression that this could be used instead of fallback: true option in the e2e test. not sure what i am missing

cmaddalozzo · 2025-06-04T20:21:28Z

hmm i was under the impression that this could be used instead of fallback: true option in the e2e test. not sure what i am missing

ah, yes that will work. The part that can't be tested yet is retrying across priorities in a dynamic way.

mathetake · 2025-06-04T20:40:05Z

cool, could you go ahead and make the change like the below;

diff --git a/examples/provider_fallback/base.yaml b/examples/provider_fallback/base.yaml
index b695ae4..ea3c5bc 100644
--- a/examples/provider_fallback/base.yaml
+++ b/examples/provider_fallback/base.yaml
@@ -41,8 +41,11 @@ spec:
               name: x-ai-eg-model
               value: us.meta.llama3-2-1b-instruct-v1:0
       backendRefs:
-        - name: provider-fallback-always-failing-upstream  # This is the primary backend and trying to speak TLS, which always fails.
+        - name: provider-fallback-always-failing-upstream
+          # This is the primary backend and trying to speak TLS, which always fails.
+          priority: 0
         - name: provider-fallback-aws
+          priority: 1
 ---
 apiVersion: aigateway.envoyproxy.io/v1alpha1
 kind: AIServiceBackend
@@ -80,8 +83,6 @@ metadata:
   name: provider-fallback-aws
   namespace: default
 spec:
-  # Indicate that this backend is a fallback backend, meaning that it will only be used if the primary backend fails.
-  fallback: true
   endpoints:
     - fqdn:
         hostname: bedrock-runtime.us-east-1.amazonaws.com

i believe this should work and then we are good to go (i think we can make this part of v0.2)

extension server. Signed-off-by: Curtis Maddalozzo <cmaddalozzo@bloomberg.net>

Signed-off-by: Curtis Maddalozzo <cmaddalozzo@bloomberg.net>

Signed-off-by: Dan Sun <dsun20@bloomberg.net>

yuzisun · 2025-06-04T23:02:15Z

cool, could you go ahead and make the change like the below;

diff --git a/examples/provider_fallback/base.yaml b/examples/provider_fallback/base.yaml
index b695ae4..ea3c5bc 100644
--- a/examples/provider_fallback/base.yaml
+++ b/examples/provider_fallback/base.yaml
@@ -41,8 +41,11 @@ spec:
               name: x-ai-eg-model
               value: us.meta.llama3-2-1b-instruct-v1:0
       backendRefs:
-        - name: provider-fallback-always-failing-upstream  # This is the primary backend and trying to speak TLS, which always fails.
+        - name: provider-fallback-always-failing-upstream
+          # This is the primary backend and trying to speak TLS, which always fails.
+          priority: 0
         - name: provider-fallback-aws
+          priority: 1
 ---
 apiVersion: aigateway.envoyproxy.io/v1alpha1
 kind: AIServiceBackend
@@ -80,8 +83,6 @@ metadata:
   name: provider-fallback-aws
   namespace: default
 spec:
-  # Indicate that this backend is a fallback backend, meaning that it will only be used if the primary backend fails.
-  fallback: true
   endpoints:
     - fqdn:
         hostname: bedrock-runtime.us-east-1.amazonaws.com

i believe this should work and then we are good to go (i think we can make this part of v0.2)

@mathetake I updated, going to update my doc with the same change.

mathetake

Awesome

mathetake · 2025-06-04T23:17:03Z

examples/provider_fallback/base.yaml

        - name: provider-fallback-always-failing-upstream  # This is the primary backend and trying to speak TLS, which always fails.
+          priority: 0
        - name: provider-fallback-aws
+          priority: 1


Can you also delete the fallback: true below as it's overridden by this

Yes I deleted the fallback flag below

**Commit Message** Add priority field to AIGatewayRouteRuleBackendRef. This will set priority on the underlying LocalityLB endpoint in Envoy. **Related Issues/PRs (if applicable)** N/A **Special notes for reviewers (if applicable)** In order to get the Envoy Gateway to reconcile the underlying `HTTPRoute` when the priorities in the `AIGatewayRoute` spec change we need to add an annotation. This annotation just store the priorities for each rule. This ensure we only trigger reconciliation when the priorities actually change. --------- Signed-off-by: Curtis Maddalozzo <cmaddalozzo@bloomberg.net> Signed-off-by: Dan Sun <dsun20@bloomberg.net> Co-authored-by: Dan Sun <dsun20@bloomberg.net>

cmaddalozzo force-pushed the add-backend-ref-priority branch from eb2aa3b to 102c94b Compare June 3, 2025 19:29

cmaddalozzo changed the title ~~Add priority to backendRef. Set priority on cluster endpoints in~~ feat: add priority to backendRef. Set priority on cluster endpoints in Jun 3, 2025

cmaddalozzo changed the title ~~feat: add priority to backendRef. Set priority on cluster endpoints in~~ feat: add priority to backendRef Jun 3, 2025

cmaddalozzo marked this pull request as ready for review June 3, 2025 19:59

cmaddalozzo requested a review from a team as a code owner June 3, 2025 19:59

aabchoo reviewed Jun 3, 2025

View reviewed changes

api/v1alpha1/api.go Show resolved Hide resolved

internal/controller/ai_gateway_route.go Outdated Show resolved Hide resolved

cmaddalozzo force-pushed the add-backend-ref-priority branch from 76f2dbe to 79f2fcd Compare June 4, 2025 14:51

mathetake reviewed Jun 4, 2025

View reviewed changes

internal/controller/ai_gateway_route.go Show resolved Hide resolved

mathetake reviewed Jun 4, 2025

View reviewed changes

internal/extensionserver/extensionserver.go Show resolved Hide resolved

cmaddalozzo force-pushed the add-backend-ref-priority branch from 515f83b to 02c5208 Compare June 4, 2025 19:23

mathetake reviewed Jun 4, 2025

View reviewed changes

api/v1alpha1/api.go Show resolved Hide resolved

cmaddalozzo and others added 7 commits June 4, 2025 18:58

Add priority to backendRef. Set priority on cluster endpoints in

16d53ac

extension server. Signed-off-by: Curtis Maddalozzo <cmaddalozzo@bloomberg.net>

Add annotation to trigger reconciliation when priority changes

73826b7

Signed-off-by: Curtis Maddalozzo <cmaddalozzo@bloomberg.net>

Fix null pointer references

9954a3d

Signed-off-by: Curtis Maddalozzo <cmaddalozzo@bloomberg.net>

Add comment about overridding fallback property

cd296ed

Signed-off-by: Curtis Maddalozzo <cmaddalozzo@bloomberg.net>

Update docs

49034ae

Signed-off-by: Curtis Maddalozzo <cmaddalozzo@bloomberg.net>

Add comment on annotation usage

0c23006

Signed-off-by: Curtis Maddalozzo <cmaddalozzo@bloomberg.net>

Add priority to fallback example

9e2d702

Signed-off-by: Dan Sun <dsun20@bloomberg.net>

yuzisun force-pushed the add-backend-ref-priority branch from a2ed196 to 9e2d702 Compare June 4, 2025 23:00

mathetake approved these changes Jun 4, 2025

View reviewed changes

yuzisun merged commit 0a0ef4a into envoyproxy:main Jun 5, 2025
17 checks passed

Conversation

cmaddalozzo commented Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mathetake left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mathetake commented Jun 4, 2025

Uh oh!

Uh oh!

cmaddalozzo commented Jun 4, 2025

Uh oh!

mathetake commented Jun 4, 2025

Uh oh!

cmaddalozzo commented Jun 4, 2025

Uh oh!

mathetake commented Jun 4, 2025

Uh oh!

yuzisun commented Jun 4, 2025

Uh oh!

mathetake left a comment

Choose a reason for hiding this comment

Uh oh!

mathetake Jun 4, 2025

Choose a reason for hiding this comment

Uh oh!

yuzisun Jun 4, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

cmaddalozzo commented Jun 3, 2025 •

edited

Loading