feat(gateway-api): add ability to override target on *Route resources

[davidspek]

Description

This PR enhances Gateway API *Route (HTTPRoute, GRPCRoute, TLSRoute, TCPRoute, UDPRoute) target resolution by:

1. Allowing per-Route override of Gateway targets via external-dns.alpha.kubernetes.io/target.
2. Introducing a new per-Route strategy annotation external-dns.alpha.kubernetes.io/target-strategy controlling how Route, Gateway, and Gateway status.addresses sources are combined.

Supported strategies (and behavior):

• route-preferred (default when unset or unknown): Route targets if any, else Gateway targets if any, else Gateway addresses.
• route-only: Route targets if any, else Gateway addresses (Gateway target annotation ignored).
• gateway-only: Gateway targets if any, else Gateway addresses (Route target annotation ignored).
• merge: Union of Route + Gateway targets (deduped); if neither present, fallback to Gateway addresses.

Fallback to Gateway status.addresses always ensures a valid target set when annotations contribute none.

Backward Compatibility:

Default behavior for Routes without a strategy now aligns with typical operator expectations: a Route’s own target takes precedence if specified; otherwise existing Gateway behavior is preserved.

Fixes #4779

Checklist

• Unit tests updated
• End user documentation updated

[k8s-ci-robot]

Hi @davidspek. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[mloiseleur]

/ok-to-test
/assign @abursavich for a first review

[davidspek]

/retest

[mloiseleur]

I took some time to think about this. I understand the use case, it can give more flexibility. The use case is ok for me.

The current approach looks dangerous and quite an anti-pattern to me: as a user, when I set an empty value, I won't expect it to enable/disable something else in a parent. This annotation is already an override over the parent Gateway target.

IMHO, to override this override, you need to introduce an other annotation and/or a CLI flag. Something like external-dns.alpha.kubernetes.io/skip-gateway-target: true or external-dns.alpha.kubernetes.io/force-gateway-address: true.

I'll let others reviewers share their thoughts, they may see it differently.

[davidspek]

My thinking was that the annotation on the routes would override the annotation on the gateway. As such, setting an empty string would be as if you set an empty string on the Gateway. However, looking at it from another perspective a user might not see the route annotation as an override of the annotation on the gateway and expect an empty string to be the same as if there was no annotation at all. I guess it depends on how you look at it.

Admittedly I would find it a bit strange to explicitly be setting an empty string value on a route if you don't intend it to be an override for the annotation on the gateway.

Personally I'm always in favor of the simplest solution which I think the current approach is, but I'd also be fine in adding a second annotation. I think using a CLI flag would reduce the flexibility. The question then become, what should this annotation be?

[mloiseleur]

@abursavich Wdyt ?

[davidspek]

@mloiseleur Is there possibly somebody else that could also provide a review?

[davidspek]

@mloiseleur Any new on this?

[ivankatliarchuk]

/label tide/merge-method-squash

[davidspek]

@ivankatliarchuk Would you be able to add a review here?

[ivankatliarchuk]

Hi. Gateway CRDs is not something I'm familiar with. Too many legacy projects, so need some time to understand in full Gateway API. Before to review code, would you be able to share manifests files to reproduce the setup, so I could apply in my local cluster and results of your smoke tests as well?

[davidspek]

@ivankatliarchuk Sorry for the slow response, I've been juggling a lot of projects at work. You most likely would need to deploy some kind of controller that implements Gateway API, such as Istio. They have a good guide for getting started here.

Just copy/pasting their examples here to make the workflow clear:

A Gateway is created that specifies an alternative target

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: gateway
  namespace: istio-ingress
  annotations:
    external-dns.alpha.kubernetes.io/target: 4.3.2.1
spec:
  gatewayClassName: istio
  listeners:
  - name: default
    hostname: "*.example.com"
    port: 80
    protocol: HTTP
    allowedRoutes:
      namespaces:
        from: All
status:
  addresses:
  - type: IPAddress
    value: 1.2.3.4

A HTTPRoute (or other *Route type) without a target annotation would use the target from the Gateway. In this case 4.3.2.1

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: http
  namespace: default
spec:
  parentRefs:
  - name: gateway
    namespace: istio-ingress
  hostnames: ["httpbin.example.com"]
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /get
    backendRefs:
    - name: httpbin
      port: 8000

If a HTTPRoute has its own target annotation this value will be used.

So the following HTTPRoute would have the target 2.3.4.5.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: http-route-target
  namespace: default
  annotations:
    external-dns.alpha.kubernetes.io/target: 2.3.4.5
spec:
  parentRefs:
  - name: gateway
    namespace: istio-ingress
  hostnames: ["httpbin-route-target.example.com"]
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /get
    backendRefs:
    - name: httpbin
      port: 8000

And the following HTTPRoute would have the target 1.2.3.4, since the target annotation is "". The same result you would get if the Gateway had this target annotation.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: http-override
  namespace: default
  annotations:
    external-dns.alpha.kubernetes.io/target: ""
spec:
  parentRefs:
  - name: gateway
    namespace: istio-ingress
  hostnames: ["httpbin-override.example.com"]
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /get
    backendRefs:
    - name: httpbin
      port: 8000

[ivankatliarchuk]

My recent testing/review supports the discussion found at with #5319 (comment). With slightly a difference, not to add new annotations, but to support something very similar to --annotation-filter for override default behaviour.

My reasoning is that:

• One user might opt to implement an override.
• Another might choose not to apply an override while still retaining the original annotations.
• The core idea is that users should have flexibility

Therefore, I advocate for implementing both a "soft" override (activated by presence of specific annotation) and a "force" override (beneficial for critical SRE, platform, or infrastructure migrations).

[davidspek]

@ivankatliarchuk Could you maybe explain in a little more detail what you have in mind exactly? Then I can get that implemented so this PR hopefully doesn't stay open for too long.

[ivankatliarchuk]

It's tough for me to suggest a solution without potentially disrupting someone's workflow, as I haven't personally encountered a use case that requires this yet.

I'm currently working on a proposal for annotation improvements (see #5080), and I don't believe an empty target annotation is a desirable feature. The binary approach is easier to understand and support: the target annotation is either set with a value or it's not set at all.

I'd suggest sharing the pros and cons of each approach you're considering. Alternatively, you might already have a preferred idea based on our discussion.

[nichcuta]

Any plans to include this change in the upcoming 0.18 release?

[daegalus]

The usecase is if you have multiple HTTPRoutes on a single gateway for use with subdomains.

For example, the Gateway handles all *.example.com hostnames. I can set it to a default of my server's cname i have set.

For some services I run need to be behind a CDN, they need a different target. Instead of having to maintain multiple gateways (which is cumbersome, and not easily automateable), I set the annotation on the HTTPRoute. This way those specific services get set to a CDN target that they points to the appropriate cname that hits the server.

Assuming 1 gateway per domain is a bad assumption. Becuase you can have 64 domains technically per gateway, including wildcards, and an unlimited number of HTTPRoutes with Hostnames that bind to that gateway. You should be able to control all annotations on a per HTTPRoute/GRPCRoute, then looking up to the Gateway for gateway wide defaults (like TTLs or anything else).

Right now this doesn't work and is a serious blocker for using external DNS with gateways. Without the annotation, it just sets A/AAAA records directly to the servers IP, removing them from the CDN/Load Balancer

So this PR is really important for me to hopefully be merged soon.

[mloiseleur]

@daegalus @nichcuta @davidspek Following @ivankatliarchuk and my review, this PR won't be merged "as is".

The use case is ok, the current approach / implementation is not. Feel free to:

• Open a proposal to discuss other approach
• Open a PR with a different implementation
• Rework this PR with a different approach

[davidspek]

@mloiseleur @ivankatliarchuk Sorry for the long time it's taken for me to get back to this. I've revised the implementation and documentation and updated the PR description.

Please let me know what your thoughts are regarding this approach.

[davidspek]

/retest

[davidspek]

/retest

[davidspek]

@mloiseleur @ivankatliarchuk It seems like the unit tests are now failing because of a race condition in the tests that should be resolved by #5841. Other than that, how does the new implementation look?

[mloiseleur]

@davidspek: The PR to fix the race condition has been merged.

Other than that, how does the new implementation look?

It looks way better to me. @davidspek: Has it been tested on a real cluster ?

@abursavich: Do you think you can take a look and review this PR ? This approach is more flexible and may solve some configuration challenges users can have with Gateway API on dns. But maybe I missed something.

[abursavich]

My general position is that perceived deficiencies in the expressiveness of Gateway API should be addressed in the Gateway API spec, wherever possible, instead of hacking them into ExternalDNS. They're very welcoming, have regular meetings, and a well-defined enhancements proposal process. Adding features to override Gateway API specifications in ExternalDNS makes the implementation more complicated, unpredictable, and brittle to future upstream changes.

If target overrides on Routes are necessary, then the preference of Route override -> Gateway override -> Gateway address (e.g. route-preferred) makes sense to me. The complexity of offering the other strategies doesn't seem necessary to me and may not play well with the evolution of the Gateway API.

Consider the explosion of strategies that might be added to support the currently experimental GEP-1713 ListenerSets, which adds ListenerSets as another link in the chain and possible place to add an override annotation between the Route and Gateway. If ListenerSets were standard at this point, I would suggest that the preference of ListenerSet override -> Gateway override -> Gateway address makes the most sense and Route overrides shouldn't be supported.

From the perspective of Gateway API's design for roles and personas, I think IPs should be controlled by Infrastructure Providers or Cluster Operators that own Gateways and Listeners not Application Developers that own Routes. I don't understand the use case for a Route needing a different IP address. That sounds like a different Gateway (or Listener) to me. All of that said, I'm okay with supporting the implicit route-preferred strategy if it's really needed, but I would need convincing for the other strategies.

[abursavich]

I just saw @daegalus's comment:

For example, the Gateway handles all *.example.com hostnames. I can set it to a default of my server's cname i have set.

For some services I run need to be behind a CDN, they need a different target. Instead of having to maintain multiple gateways (which is cumbersome, and not easily automateable), I set the annotation on the HTTPRoute. This way those specific services get set to a CDN target that they points to the appropriate cname that hits the server.

My preference to support this would be to add an annotation that causes ExternalDNS to ignore the HTTPRoute (or any Source for that matter) so the user can create a DNSEndpoint for the CDN that won't fight with the HTTPRoute's endpoints. That way there's an escape hatch for anyone that wants to do something different, without needing to bake every possible special case into every Source.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

• Mark this PR as fresh with /remove-lifecycle stale
• Close this PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-ci-robot]

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[iAnomaly]

I just saw @daegalus's comment:

For example, the Gateway handles all *.example.com hostnames. I can set it to a default of my server's cname i have set.
For some services I run need to be behind a CDN, they need a different target. Instead of having to maintain multiple gateways (which is cumbersome, and not easily automateable), I set the annotation on the HTTPRoute. This way those specific services get set to a CDN target that they points to the appropriate cname that hits the server.

My preference to support this would be to add an annotation that causes ExternalDNS to ignore the HTTPRoute (or any Source for that matter) so the user can create a DNSEndpoint for the CDN that won't fight with the HTTPRoute's endpoints. That way there's an escape hatch for anyone that wants to do something different, without needing to bake every possible special case into every Source.

FWIW I was able to achieve ignoring a Gateway resource via the existing --gateway-label-filter flag and looking through the existing gateway source code it looks like *Routes would be ignored with the --label-filter flag.

I did find it confusing that --gateway-label-filter is a separate flag versus the logic following --label-filter for both Gateway kinds AND *Route kinds. Maybe this is from the historical confusion around annotation/label inheritance/layering/merging of parents/Gateways and children/*Routes though.

Thank you @ivankatliarchuk @mloiseleur @lexfrei for all of your work on the Gateway API annotation/label placement clarity and @davidspek for this target PR!

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

• Mark this PR as fresh with /remove-lifecycle rotten
• Close this PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

• Reopen this PR with /reopen
• Mark this PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closed this PR.

Details

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

• Reopen this PR with /reopen
• Mark this PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.