From 056c52f140726fbce2fb0b07dab21908fd66cb03 Mon Sep 17 00:00:00 2001 From: dprotaso Date: Tue, 21 Mar 2023 21:06:34 -0400 Subject: [PATCH 01/36] initial commit of GEP 1713 --- geps/gep-1713.md | 103 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 103 insertions(+) create mode 100644 geps/gep-1713.md diff --git a/geps/gep-1713.md b/geps/gep-1713.md new file mode 100644 index 0000000000..eabff3ceb5 --- /dev/null +++ b/geps/gep-1713.md @@ -0,0 +1,103 @@ +# GEP-1713: Standard Mechanism to Merge Multiple Gateways + +* Issue: [#1713/kubernetes-sigs/gateway-api/issues/1713) +* Status: Provisional + +(See status definitions [here](overview.md#status).) + +## TLDR + +The Gateway Resource is a contention point since it is the only place to attach listeners with certificates. We propose a mechanism to allow distinct Gateway resources to be logically merged. + +## Goals + +- Define a mechanic to merge multiple Gateways (logically) +- Define a set of acceptable properties that can be merged and their semantics + +## Non-Goals + +- Apply a Gateway resource onto N distinct gateways (one to many) + +## Introduction + +Knative generates on demand per-service certificates using HTTP-01 challenges. There can be O(1000) Knative Services in the cluster which means we have O(1000) distinct certificates. Thus updating a single Gateway resource with this many certificates is a contention point and inhibits horizontal scaling of our controllers. + +The spec currently has language to indicate implemenations `MAY` merge Gateways resources but the mechanic isn't defined. +https://github.com/kubernetes-sigs/gateway-api/blob/541e9fc2b3c2f62915cb58dc0ee5e43e4096b3e2/apis/v1beta1/gateway_types.go#L76-L78 + +## Future Requirement Level + +We want to keep this API very simple so that the merging requirement level could increase from `MAY` to `MUST` + +## API + +```yaml +apiVersion: gateway.networking.k8s.io/v1beta1 +kind: Gateway +metadata: + name: parent-gateway +spec: + gatewayClassName: example + listeners: + - name: default + port: 80 + protocol: HTTP +--- +apiVersion: gateway.networking.k8s.io/v1beta1 +kind: Gateway +metadata: + name: child-gateway + labels: + gateway.networking.k8s.io/parent-gateway: parent-gateway +spec: + gatewayClassName: example + parentGateway: parent-gateway + listeners: + - name: metrics + port: 8080 + protocol: HTTP +``` + +With this configuration, an implementation: +* This introduces a new concept in Gateway merging of a "primary" Gateway. +* `MUST` treat the `parentGateway` as having the concatenated list of all listeners from merged Gateways. + This means, for example, that a Service with port 80 and 443 may be created in this example. +* This makes it explicit which is the primary gateway and which are attached to it. +* Secondary Gateways cannot configure `addresses` and must have the same `gatewayClassName` as the primary Gateway they are attached to. +* Gateways can only reference a primary Gateway. That is, we cannot have a chain of references. +* A common webhook will default the `gateway.networking.k8s.io/parent-gateway` label to make it easy for implementations to label selector lookups +* Each `Gateway` will only list listeners in their status if there is a corresponding entry in their spec. + +## Alternatives + +#### New Resource +A `GatewayListener` resource could be a simpler solution as we would not have to set required fields (ie. gatewayClassName) + +``` +apiVersion: gateway.networking.k8s.io/v1beta1 +kind: GatewayListener +metadata: + name: listener + labels: + gateway.networking.k8s.io/parent-gateway: parent-gateway +spec: + gateway: parent-gateway + listeners: + - name: metrics + port: 8080 + protocol: HTTP +status: ... +``` + +#### Use of the `gateway.networking.k8s.io/parent-gateway` label + +We could drop the `parentGateway` in the `spec` and just leverage the use of a label to signal a parent-child relationship. + +## References +Mentioned in Prior GEPs: +- https://github.com/kubernetes-sigs/gateway-api/pull/1757 + +Prior Discussions: +- https://github.com/kubernetes-sigs/gateway-api/discussions/1248 +- https://github.com/kubernetes-sigs/gateway-api/discussions/1246 + From b0da0610389272fac24f76dda7318b683740eb6f Mon Sep 17 00:00:00 2001 From: dprotaso Date: Tue, 28 Mar 2023 19:23:25 -0400 Subject: [PATCH 02/36] Flush out the GEP --- geps/gep-1713.md | 153 +++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 134 insertions(+), 19 deletions(-) diff --git a/geps/gep-1713.md b/geps/gep-1713.md index eabff3ceb5..8a5efb3ca7 100644 --- a/geps/gep-1713.md +++ b/geps/gep-1713.md @@ -1,11 +1,11 @@ # GEP-1713: Standard Mechanism to Merge Multiple Gateways -* Issue: [#1713/kubernetes-sigs/gateway-api/issues/1713) +* Issue: [#1713](/kubernetes-sigs/gateway-api/issues/1713) * Status: Provisional (See status definitions [here](overview.md#status).) -## TLDR +## tl;dr The Gateway Resource is a contention point since it is the only place to attach listeners with certificates. We propose a mechanism to allow distinct Gateway resources to be logically merged. @@ -25,12 +25,101 @@ Knative generates on demand per-service certificates using HTTP-01 challenges. T The spec currently has language to indicate implemenations `MAY` merge Gateways resources but the mechanic isn't defined. https://github.com/kubernetes-sigs/gateway-api/blob/541e9fc2b3c2f62915cb58dc0ee5e43e4096b3e2/apis/v1beta1/gateway_types.go#L76-L78 -## Future Requirement Level +## API -We want to keep this API very simple so that the merging requirement level could increase from `MAY` to `MUST` +We propose adding a new `infrastructure` stanza to the `spec` field of a Gateway. Within `infrastructure` there is an `attachTo` field of type `LocalObjectReference`. Although the use of `LocalObjectReference` allows users to attach to any `kind`, this GEP only defines the behaviour of attaching a Gateway to another Gateway. -## API +See [GEP-1867](https://github.com/kubernetes-sigs/gateway-api/pull/1868) for more use cases of `infrastructure`. + +#### Go + +```go +type GatewaySpec struct { + // Infrastructure defines infrastructure level attributes about this Gateway instance. + Infrastructure GatewayInfrastructure `json:"infrastructure"` + // ... +} + +type GatewayInfrastructure struct { + // AttachTo allows the Gateway to associate itself with another resource. + // A common scenario is to reference another Gateway which marks + // this Gateway a child of another. + AttachTo LocalObjectReference `json:"attachTo"` +} +``` + +#### YAML +```yaml +apiVersion: gateway.networking.k8s.io/v1beta1 +kind: Gateway +metadata: + name: child-gateway + labels: + gateway.networking.k8s.io/parent-gateway: parent-gateway +spec: + gatewayClassName: example + infrastructure: + attachTo: + name: parent-gateway + kind: Gateway + group: gateway.networking.k8s.io + listeners: + - name: metrics + port: 8080 + protocol: HTTP +``` + +### Semantics + + +#### Gateway Attaching + +Gateways that _do not_ reference another Gateway (using `spec.infrastructure.attachTo`) are considered "primary" Gateways. A "child" Gateway is one where `spec.infrastructure.attachTo` field is set to a "primary" Gateway. A "sibling" is a Gateway that shares a "parent" with another "child" Gateway. + + +Failure to attach a "child" Gateway to another MUST result in the "child" Gateway setting their `Accepted` condition to `False`. This MUST happen in the following scenarios: +- Attaching to another "child" Gateway +- Attaching to oneself (self-referential) +- Attaching to a non-existing Gateway +- Attaching to a Gateway with a different `GatewayClassName` + +#### Route Attaching + +Routes MUST be able to specify a "child" Gateways as a `parentRef` and make use of the fields in `ParentReference` to help target a specific listener. + + +```yaml +apiVersion: gateway.networking.k8s.io/v1beta1 +kind: HTTPRoute +metadata: + name: httproute-example +spec: + parentRefs: + - name: acme-lb + sectionName: https +``` + +#### `parent-gateway` label + +Implementations MUST label "child" Gateways with the `gateway.networking.k8s.io/parent-gateway` label. The value of the label is the name of the "parent" Gateway and it MUST be equivalent to the value `spec.infrastructure.attachTo.name` + +Users MUST NOT be able to edit this field. + +#### Merging Spec Fields + + +#### GatewayClassName + +The `GatewayClassName` MUST be set on "child" and "primary" Gateways and be equivalent. + +#### Listeners + +Implementations MUST treat the "primary" Gateway as having the concatenated list of all listeners from "child" Gateways. + +Validation of this list of listeners is subject to the discussion here: https://github.com/kubernetes-sigs/gateway-api/issues/1842 + +eg. ```yaml apiVersion: gateway.networking.k8s.io/v1beta1 kind: Gateway @@ -39,9 +128,9 @@ metadata: spec: gatewayClassName: example listeners: - - name: default + - name: HTTP port: 80 - protocol: HTTP + protocol: HTTP --- apiVersion: gateway.networking.k8s.io/v1beta1 kind: Gateway @@ -51,22 +140,47 @@ metadata: gateway.networking.k8s.io/parent-gateway: parent-gateway spec: gatewayClassName: example - parentGateway: parent-gateway + infrastructure: + attachTo: + name: parent-gateway + kind: Gateway + group: gateway.networking.k8s.io listeners: - name: metrics port: 8080 - protocol: HTTP + protocol: HTTP ``` -With this configuration, an implementation: -* This introduces a new concept in Gateway merging of a "primary" Gateway. -* `MUST` treat the `parentGateway` as having the concatenated list of all listeners from merged Gateways. - This means, for example, that a Service with port 80 and 443 may be created in this example. -* This makes it explicit which is the primary gateway and which are attached to it. -* Secondary Gateways cannot configure `addresses` and must have the same `gatewayClassName` as the primary Gateway they are attached to. -* Gateways can only reference a primary Gateway. That is, we cannot have a chain of references. -* A common webhook will default the `gateway.networking.k8s.io/parent-gateway` label to make it easy for implementations to label selector lookups -* Each `Gateway` will only list listeners in their status if there is a corresponding entry in their spec. +With this configuration, the realized "primary" Gateway should listen on port `80` and `8080`. + + +#### Addresses + +Merging `Addresses` is not supported and the "child" Gateway's `Addresses` MUST be empty. + + +### Status Fields + + +#### Addresses + +The list of `Addresses` that appear in the status of the "child" Gateway MUST be the same as the "parent" Gateway. + +#### Listeners + +The "child" Gateway MUST provide a status for each Listener defined in their specification. Listeners defined in "sibling" and the "parent" Gateway SHOULD NOT appear on a "child" Gateway. + +#### Conditions + +"Child" Gateways MUST support the same types of status conditions as "parent" Gateways. + + +## Future Goals + +### Requirement Level + +We want to keep this API very simple so that the merging requirement level could increase from `MAY` to `MUST` + ## Alternatives @@ -91,7 +205,7 @@ status: ... #### Use of the `gateway.networking.k8s.io/parent-gateway` label -We could drop the `parentGateway` in the `spec` and just leverage the use of a label to signal a parent-child relationship. +Use of the label could be used to select child gateways vs using `spec.infrastructure.attachTo` ## References Mentioned in Prior GEPs: @@ -101,3 +215,4 @@ Prior Discussions: - https://github.com/kubernetes-sigs/gateway-api/discussions/1248 - https://github.com/kubernetes-sigs/gateway-api/discussions/1246 + From 3f50cea4469ba551df449ce579eb8a59eda40f2e Mon Sep 17 00:00:00 2001 From: dprotaso Date: Tue, 28 Mar 2023 19:44:32 -0400 Subject: [PATCH 03/36] fix typos --- geps/gep-1713.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/geps/gep-1713.md b/geps/gep-1713.md index 8a5efb3ca7..5a55c81110 100644 --- a/geps/gep-1713.md +++ b/geps/gep-1713.md @@ -22,7 +22,7 @@ The Gateway Resource is a contention point since it is the only place to attach Knative generates on demand per-service certificates using HTTP-01 challenges. There can be O(1000) Knative Services in the cluster which means we have O(1000) distinct certificates. Thus updating a single Gateway resource with this many certificates is a contention point and inhibits horizontal scaling of our controllers. -The spec currently has language to indicate implemenations `MAY` merge Gateways resources but the mechanic isn't defined. +The spec currently has language to indicate implementations `MAY` merge Gateways resources but the mechanic isn't defined. https://github.com/kubernetes-sigs/gateway-api/blob/541e9fc2b3c2f62915cb58dc0ee5e43e4096b3e2/apis/v1beta1/gateway_types.go#L76-L78 ## API @@ -168,7 +168,7 @@ The list of `Addresses` that appear in the status of the "child" Gateway MUST be #### Listeners -The "child" Gateway MUST provide a status for each Listener defined in their specification. Listeners defined in "sibling" and the "parent" Gateway SHOULD NOT appear on a "child" Gateway. +The "child" Gateways MUST provide a status for each Listener defined in their specification. Listeners defined in "sibling" and the "parent" Gateway SHOULD NOT appear on a "child" Gateway. #### Conditions From 09b29d162b2f6b37547de79dd9272819a265fcfd Mon Sep 17 00:00:00 2001 From: dprotaso Date: Tue, 28 Mar 2023 19:44:57 -0400 Subject: [PATCH 04/36] Accepted should trigger other conditions to be set to 'False' --- geps/gep-1713.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/geps/gep-1713.md b/geps/gep-1713.md index 5a55c81110..f8e812c4e9 100644 --- a/geps/gep-1713.md +++ b/geps/gep-1713.md @@ -84,6 +84,8 @@ Failure to attach a "child" Gateway to another MUST result in the "child" Gatewa - Attaching to a non-existing Gateway - Attaching to a Gateway with a different `GatewayClassName` +If a Gateway's `Accepted` condition is `False` then the conditions 'Programmed' and 'Ready' (if supported) MUST be set to `False`. + #### Route Attaching Routes MUST be able to specify a "child" Gateways as a `parentRef` and make use of the fields in `ParentReference` to help target a specific listener. From f4f392883c7c00d38087a933f228e159914c8288 Mon Sep 17 00:00:00 2001 From: dprotaso Date: Tue, 4 Apr 2023 18:12:13 -0400 Subject: [PATCH 05/36] Drop parent-gateway label --- geps/gep-1713.md | 14 +------------- 1 file changed, 1 insertion(+), 13 deletions(-) diff --git a/geps/gep-1713.md b/geps/gep-1713.md index f8e812c4e9..804ebdfdd6 100644 --- a/geps/gep-1713.md +++ b/geps/gep-1713.md @@ -55,8 +55,6 @@ apiVersion: gateway.networking.k8s.io/v1beta1 kind: Gateway metadata: name: child-gateway - labels: - gateway.networking.k8s.io/parent-gateway: parent-gateway spec: gatewayClassName: example infrastructure: @@ -102,12 +100,6 @@ spec: sectionName: https ``` -#### `parent-gateway` label - -Implementations MUST label "child" Gateways with the `gateway.networking.k8s.io/parent-gateway` label. The value of the label is the name of the "parent" Gateway and it MUST be equivalent to the value `spec.infrastructure.attachTo.name` - -Users MUST NOT be able to edit this field. - #### Merging Spec Fields @@ -138,8 +130,6 @@ apiVersion: gateway.networking.k8s.io/v1beta1 kind: Gateway metadata: name: child-gateway - labels: - gateway.networking.k8s.io/parent-gateway: parent-gateway spec: gatewayClassName: example infrastructure: @@ -194,8 +184,6 @@ apiVersion: gateway.networking.k8s.io/v1beta1 kind: GatewayListener metadata: name: listener - labels: - gateway.networking.k8s.io/parent-gateway: parent-gateway spec: gateway: parent-gateway listeners: @@ -207,7 +195,7 @@ status: ... #### Use of the `gateway.networking.k8s.io/parent-gateway` label -Use of the label could be used to select child gateways vs using `spec.infrastructure.attachTo` +Use of a label (ie. `gateway.networking.k8s.io/parent-gateway: name`) could be used to select child gateways vs using `spec.infrastructure.attachTo` ## References Mentioned in Prior GEPs: From 4c5b38b0556cb0e754a69326d89b8cffa6077463 Mon Sep 17 00:00:00 2001 From: dprotaso Date: Tue, 4 Apr 2023 18:23:08 -0400 Subject: [PATCH 06/36] add clause that child listeners should not appear on the parent --- geps/gep-1713.md | 1 + 1 file changed, 1 insertion(+) diff --git a/geps/gep-1713.md b/geps/gep-1713.md index 804ebdfdd6..685d2150cd 100644 --- a/geps/gep-1713.md +++ b/geps/gep-1713.md @@ -161,6 +161,7 @@ The list of `Addresses` that appear in the status of the "child" Gateway MUST be #### Listeners The "child" Gateways MUST provide a status for each Listener defined in their specification. Listeners defined in "sibling" and the "parent" Gateway SHOULD NOT appear on a "child" Gateway. +Listeners in "child" Gateways SHOULD NOT appear on the "parent" Gateway. #### Conditions From cbfea8c2c93f34fe2636f00759099ffb04add450 Mon Sep 17 00:00:00 2001 From: dprotaso Date: Tue, 4 Apr 2023 18:31:18 -0400 Subject: [PATCH 07/36] update example of attaching to a child gateway --- geps/gep-1713.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/geps/gep-1713.md b/geps/gep-1713.md index 685d2150cd..c06595a795 100644 --- a/geps/gep-1713.md +++ b/geps/gep-1713.md @@ -96,8 +96,8 @@ metadata: name: httproute-example spec: parentRefs: - - name: acme-lb - sectionName: https + - name: child-gateway + sectionName: metrics ``` #### Merging Spec Fields From 832fa8f22dd4d06a830cf3e6a9d2c0a87e81a3e6 Mon Sep 17 00:00:00 2001 From: dprotaso Date: Tue, 4 Apr 2023 18:47:38 -0400 Subject: [PATCH 08/36] adjust invalid scenarios and allow for rejection via webhook --- geps/gep-1713.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/geps/gep-1713.md b/geps/gep-1713.md index c06595a795..d67fedb52c 100644 --- a/geps/gep-1713.md +++ b/geps/gep-1713.md @@ -75,15 +75,18 @@ spec: Gateways that _do not_ reference another Gateway (using `spec.infrastructure.attachTo`) are considered "primary" Gateways. A "child" Gateway is one where `spec.infrastructure.attachTo` field is set to a "primary" Gateway. A "sibling" is a Gateway that shares a "parent" with another "child" Gateway. +Failure to attach a "child" Gateway to another MUST result in the "child" Gateway setting their `Accepted` condition to `False`. -Failure to attach a "child" Gateway to another MUST result in the "child" Gateway setting their `Accepted` condition to `False`. This MUST happen in the following scenarios: -- Attaching to another "child" Gateway -- Attaching to oneself (self-referential) +This MUST happen in the following scenarios: - Attaching to a non-existing Gateway -- Attaching to a Gateway with a different `GatewayClassName` If a Gateway's `Accepted` condition is `False` then the conditions 'Programmed' and 'Ready' (if supported) MUST be set to `False`. +Invalid Gateways MUST be rejected by the implementation during creation or update. This MUST happen in the following scenarios: +- Attaching to another "child" Gateway +- Attaching to oneself (self-referential) +- Attaching to a Gateway with a different `GatewayClassName` + #### Route Attaching Routes MUST be able to specify a "child" Gateways as a `parentRef` and make use of the fields in `ParentReference` to help target a specific listener. From 492eff7fbc3f45937ddca23b5b3328e573c43ae3 Mon Sep 17 00:00:00 2001 From: dprotaso Date: Tue, 4 Apr 2023 18:57:32 -0400 Subject: [PATCH 09/36] flush out listener precedence and conflict semantics --- geps/gep-1713.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/geps/gep-1713.md b/geps/gep-1713.md index d67fedb52c..ea3bf36ef0 100644 --- a/geps/gep-1713.md +++ b/geps/gep-1713.md @@ -148,6 +148,16 @@ spec: With this configuration, the realized "primary" Gateway should listen on port `80` and `8080`. +##### Listener Precendence + +Gateway Listeners should be merged using the following precedence: +- "primary" Gateway +- "child" Gateway ordered by creation time (oldest first) +- "child" Gateway ordered alphabetical by “{namespace}/{name}”. + +##### Listener Conflict + +A "child" Gateway's listener status MUST set the condition `Conflicted` to `True` if it unable to merge with a "parent" Gateway. #### Addresses From bc245cdc0b39bc79b94a9991a904aa62dca98d44 Mon Sep 17 00:00:00 2001 From: dprotaso Date: Thu, 4 May 2023 09:48:17 -0400 Subject: [PATCH 10/36] fix header indentation --- geps/gep-1713.md | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/geps/gep-1713.md b/geps/gep-1713.md index ea3bf36ef0..bf87f1497d 100644 --- a/geps/gep-1713.md +++ b/geps/gep-1713.md @@ -70,7 +70,6 @@ spec: ### Semantics - #### Gateway Attaching Gateways that _do not_ reference another Gateway (using `spec.infrastructure.attachTo`) are considered "primary" Gateways. A "child" Gateway is one where `spec.infrastructure.attachTo` field is set to a "primary" Gateway. A "sibling" is a Gateway that shares a "parent" with another "child" Gateway. @@ -105,12 +104,11 @@ spec: #### Merging Spec Fields - -#### GatewayClassName +##### GatewayClassName The `GatewayClassName` MUST be set on "child" and "primary" Gateways and be equivalent. -#### Listeners +##### Listeners Implementations MUST treat the "primary" Gateway as having the concatenated list of all listeners from "child" Gateways. @@ -148,18 +146,18 @@ spec: With this configuration, the realized "primary" Gateway should listen on port `80` and `8080`. -##### Listener Precendence +###### Listener Precendence Gateway Listeners should be merged using the following precedence: - "primary" Gateway - "child" Gateway ordered by creation time (oldest first) - "child" Gateway ordered alphabetical by “{namespace}/{name}”. -##### Listener Conflict +###### Listener Conflict A "child" Gateway's listener status MUST set the condition `Conflicted` to `True` if it unable to merge with a "parent" Gateway. -#### Addresses +##### Addresses Merging `Addresses` is not supported and the "child" Gateway's `Addresses` MUST be empty. From 951081e537221b75a5d5ee8fd81dfb5397c839b5 Mon Sep 17 00:00:00 2001 From: dprotaso Date: Thu, 4 May 2023 09:56:37 -0400 Subject: [PATCH 11/36] Add a note that a child Gateway can attach only to other Gateways in the same namespace --- geps/gep-1713.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/geps/gep-1713.md b/geps/gep-1713.md index bf87f1497d..2b88c571ff 100644 --- a/geps/gep-1713.md +++ b/geps/gep-1713.md @@ -72,7 +72,7 @@ spec: #### Gateway Attaching -Gateways that _do not_ reference another Gateway (using `spec.infrastructure.attachTo`) are considered "primary" Gateways. A "child" Gateway is one where `spec.infrastructure.attachTo` field is set to a "primary" Gateway. A "sibling" is a Gateway that shares a "parent" with another "child" Gateway. +Gateways that _do not_ reference another Gateway (using `spec.infrastructure.attachTo`) are considered "primary" Gateways. A "child" Gateway is one where `spec.infrastructure.attachTo` field is set to a "primary" Gateway. A "sibling" is a Gateway that shares a "parent" with another "child" Gateway. A "child" Gateway can only attach to Gateways in the same namespace. Failure to attach a "child" Gateway to another MUST result in the "child" Gateway setting their `Accepted` condition to `False`. From aa1897556bd196cb90fba7d9b7ec6b3aea574cbf Mon Sep 17 00:00:00 2001 From: dprotaso Date: Thu, 4 May 2023 10:02:43 -0400 Subject: [PATCH 12/36] define parent gateway and update correct usage of primary vs parent --- geps/gep-1713.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/geps/gep-1713.md b/geps/gep-1713.md index 2b88c571ff..7ecff1ea84 100644 --- a/geps/gep-1713.md +++ b/geps/gep-1713.md @@ -72,7 +72,7 @@ spec: #### Gateway Attaching -Gateways that _do not_ reference another Gateway (using `spec.infrastructure.attachTo`) are considered "primary" Gateways. A "child" Gateway is one where `spec.infrastructure.attachTo` field is set to a "primary" Gateway. A "sibling" is a Gateway that shares a "parent" with another "child" Gateway. A "child" Gateway can only attach to Gateways in the same namespace. +Gateways that _do not_ reference another Gateway (using `spec.infrastructure.attachTo`) are considered "primary" Gateways. A "child" Gateway is one where `spec.infrastructure.attachTo` field is set to a "primary" Gateway - making the "primary" Gateway a "parent". A "sibling" is a Gateway that shares a "parent" with another "child" Gateway. A "child" Gateway can only attach to Gateways in the same namespace. Failure to attach a "child" Gateway to another MUST result in the "child" Gateway setting their `Accepted` condition to `False`. @@ -106,11 +106,11 @@ spec: ##### GatewayClassName -The `GatewayClassName` MUST be set on "child" and "primary" Gateways and be equivalent. +The `GatewayClassName` MUST be set on "child" and "parent" Gateways and be equivalent. ##### Listeners -Implementations MUST treat the "primary" Gateway as having the concatenated list of all listeners from "child" Gateways. +Implementations MUST treat the "parent" Gateway as having the concatenated list of all listeners from itself and "child" Gateways. Validation of this list of listeners is subject to the discussion here: https://github.com/kubernetes-sigs/gateway-api/issues/1842 @@ -149,13 +149,13 @@ With this configuration, the realized "primary" Gateway should listen on port `8 ###### Listener Precendence Gateway Listeners should be merged using the following precedence: -- "primary" Gateway +- "parent" Gateway - "child" Gateway ordered by creation time (oldest first) - "child" Gateway ordered alphabetical by “{namespace}/{name}”. ###### Listener Conflict -A "child" Gateway's listener status MUST set the condition `Conflicted` to `True` if it unable to merge with a "parent" Gateway. +A "child" Gateway's listener status MUST set the condition `Conflicted` to `True` if it unable to merge with a "primary" Gateway. ##### Addresses From ed2c90bbce7d3a28b234c4bb60a36407e2730bd4 Mon Sep 17 00:00:00 2001 From: Dave Protasowski Date: Wed, 2 Aug 2023 14:59:03 -0400 Subject: [PATCH 13/36] Define validation semantics on the concatenated list of listeners tl;dr - it's the same as if the listeners were part of a single Gateway --- geps/gep-1713.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/geps/gep-1713.md b/geps/gep-1713.md index 7ecff1ea84..f1b5af6a6a 100644 --- a/geps/gep-1713.md +++ b/geps/gep-1713.md @@ -112,7 +112,7 @@ The `GatewayClassName` MUST be set on "child" and "parent" Gateways and be equiv Implementations MUST treat the "parent" Gateway as having the concatenated list of all listeners from itself and "child" Gateways. -Validation of this list of listeners is subject to the discussion here: https://github.com/kubernetes-sigs/gateway-api/issues/1842 +Validation of this list of listeners MUST behave the same as if the list were part of a single "parent" Gateway. eg. ```yaml From ba5a47d89b2537a525e75e682c32270090308894 Mon Sep 17 00:00:00 2001 From: dprotaso Date: Mon, 5 Feb 2024 13:26:02 -0500 Subject: [PATCH 14/36] move GEP to dedicated directory --- geps/{gep-1713.md => gep-1713/index.md} | 0 geps/gep-1713/metadata.yaml | 7 +++++++ 2 files changed, 7 insertions(+) rename geps/{gep-1713.md => gep-1713/index.md} (100%) create mode 100644 geps/gep-1713/metadata.yaml diff --git a/geps/gep-1713.md b/geps/gep-1713/index.md similarity index 100% rename from geps/gep-1713.md rename to geps/gep-1713/index.md diff --git a/geps/gep-1713/metadata.yaml b/geps/gep-1713/metadata.yaml new file mode 100644 index 0000000000..752beaba50 --- /dev/null +++ b/geps/gep-1713/metadata.yaml @@ -0,0 +1,7 @@ +apiVersion: internal.gateway.networking.k8s.io/v1alpha1 +kind: GEPDetails +number: 1713 +name: Gateway Merging +status: Provisional +authors: + - dprotaso From e2331f0cf960bb897a5d6b65ca217413692b68a9 Mon Sep 17 00:00:00 2001 From: dprotaso Date: Tue, 6 Feb 2024 10:21:25 -0500 Subject: [PATCH 15/36] rework GEP with input from howardjohn --- geps/gep-1713/index.md | 259 ++++++++++++++++++++++++++++-------- geps/gep-1713/metadata.yaml | 1 + 2 files changed, 206 insertions(+), 54 deletions(-) diff --git a/geps/gep-1713/index.md b/geps/gep-1713/index.md index f1b5af6a6a..b9d8f8cc63 100644 --- a/geps/gep-1713/index.md +++ b/geps/gep-1713/index.md @@ -20,39 +20,145 @@ The Gateway Resource is a contention point since it is the only place to attach ## Introduction -Knative generates on demand per-service certificates using HTTP-01 challenges. There can be O(1000) Knative Services in the cluster which means we have O(1000) distinct certificates. Thus updating a single Gateway resource with this many certificates is a contention point and inhibits horizontal scaling of our controllers. +Knative generates on demand per-service certificates using HTTP-01 challenges. +There can be O(1000) Knative Services in the cluster which means we have O(1000) distinct certificates. +Thus updating a single Gateway resource with this many certificates is a contention point and inhibits horizontal scaling of our controllers. +[Istio Ambient](https://istio.io/v1.15/blog/2022/introducing-ambient-mesh/), similarly, creates a listener per Kubernetes service. + +More broadly, large scale gateway users often expose O(1000) domains, but are currently limited by the maximum of 16 `listeners`. The spec currently has language to indicate implementations `MAY` merge Gateways resources but the mechanic isn't defined. https://github.com/kubernetes-sigs/gateway-api/blob/541e9fc2b3c2f62915cb58dc0ee5e43e4096b3e2/apis/v1beta1/gateway_types.go#L76-L78 ## API -We propose adding a new `infrastructure` stanza to the `spec` field of a Gateway. Within `infrastructure` there is an `attachTo` field of type `LocalObjectReference`. Although the use of `LocalObjectReference` allows users to attach to any `kind`, this GEP only defines the behaviour of attaching a Gateway to another Gateway. +This proposal has two aspects: configuration for the parent Gateway, and configuration for the child Gateway. + +A "parent" Gateway _does not_ reference another Gateway. A parent Gateway MUST explicitly opt into merging. +This is done with a new field `allowedChildren` in the `spec.infrastructure` of a Gateway. +If, and only if, they have done so, the same Gateway is also permitted to leave `listeners` empty (currently, there is a `MinItems=1` restriction). + +A "child" Gateway references a parent Gateway. +This is done with a new field `attachTo` in the `spec.infrastructure` stanza of a Gateway. +The `attachTo` field is a new type `GatewayObjectReference`. +Although the use of `GatewayObjectReference` allows users to attach to any `kind`, this GEP only defines the behavior of attaching a Gateway to another Gateway. +A "sibling" is a Gateway that shares a parent with another child Gateway. + +Status requirements are specified [below](#status-fields). See [GEP-1867](https://github.com/kubernetes-sigs/gateway-api/pull/1868) for more use cases of `infrastructure`. + #### Go ```go -type GatewaySpec struct { - // Infrastructure defines infrastructure level attributes about this Gateway instance. - Infrastructure GatewayInfrastructure `json:"infrastructure"` - // ... +type GatewayInfrastructure struct { + // ... + + // AllowedChildren allows child objects to attach to this Gateway. + // A common scenario is to allow other objects to add listeners to this Gateway. + AllowedChildren *AllowedChildren `json:"allowedChildren,omitempty"` + + // AttachTo allows the Gateway to associate itself with another resource. + // A common scenario is to reference another Gateway which marks + // this Gateway a child of another. + AttachTo GatewayObjectReference `json:"attachTo"` } -type GatewayInfrastructure struct { - // AttachTo allows the Gateway to associate itself with another resource. - // A common scenario is to reference another Gateway which marks - // this Gateway a child of another. - AttachTo LocalObjectReference `json:"attachTo"` +// AllowedChildren defines which objects may be attached as children +type AllowedChildren struct { + // Namespaces indicates namespaces from which children may be attached to this + // Gateway. This is restricted to the namespace of this Gateway by default. + // + // Support: Core + // + // +optional + // +kubebuilder:default={from: Same} + Namespaces *ChildrenNamespaces `json:"namespaces,omitempty"` +} + +// ChildrenNamespaces indicate which namespaces Children should be selected from. +type ChildrenNamespaces struct { + // From indicates where Children will be selected for this Gateway. Possible + // values are: + // + // * All: Children in all namespaces may be used by this Gateway. + // * Selector: Children in namespaces selected by the selector may be used by + // this Gateway. + // * Same: Only Children in the same namespace may be used by this Gateway. + // + // Support: Core + // + // +optional + // +kubebuilder:default=Same + From *FromNamespaces `json:"from,omitempty"` + + // Selector must be specified when From is set to "Selector". In that case, + // only Children in Namespaces matching this Selector will be selected by this + // Gateway. This field is ignored for other values of "From". + // + // Support: Core + // + // +optional + Selector *metav1.LabelSelector `json:"selector,omitempty"` +} + +// GatewayObjectReference identifies an API object including its namespace, +// defaulting to Gateway. +type GatewayObjectReference struct { + // Group is the group of the referent. For example, "gateway.networking.k8s.io". + // When unspecified or empty string, core API group is inferred. + // + // +optional + // +kubebuilder:default="" + Group *Group `json:"group"` + + // Kind is kind of the referent. For example "Gateway". + // + // +optional + // +kubebuilder:default=Gateway + Kind *Kind `json:"kind"` + + // Name is the name of the referent. + Name ObjectName `json:"name"` + + // Namespace is the namespace of the referenced object. When unspecified, the local + // namespace is inferred. + // + // Support: Core + // + // +optional + Namespace *Namespace `json:"namespace,omitempty"` } ``` #### YAML + +Below shows an example of an end to end configuration with Gateway merging. +Here we define a parent resource, which allows children from the same namespace. +A single listener is defined in the parent. + +The child gateway attaches to this Gateway and specifies an additional listener. + ```yaml apiVersion: gateway.networking.k8s.io/v1beta1 kind: Gateway +metadata: + name: parent-gateway +spec: + gatewayClassName: example + infrastructure: + allowedChildren: + namespaces: + from: Same + listeners: + - name: common-monitoring + port: 8080 + protocol: HTTP +--- +apiVersion: gateway.networking.k8s.io/v1beta1 +kind: Gateway metadata: name: child-gateway spec: @@ -63,28 +169,32 @@ spec: kind: Gateway group: gateway.networking.k8s.io listeners: - - name: metrics - port: 8080 + - name: domain-a + hostname: a.example.com protocol: HTTP + port: 80 ``` -### Semantics - -#### Gateway Attaching - -Gateways that _do not_ reference another Gateway (using `spec.infrastructure.attachTo`) are considered "primary" Gateways. A "child" Gateway is one where `spec.infrastructure.attachTo` field is set to a "primary" Gateway - making the "primary" Gateway a "parent". A "sibling" is a Gateway that shares a "parent" with another "child" Gateway. A "child" Gateway can only attach to Gateways in the same namespace. - -Failure to attach a "child" Gateway to another MUST result in the "child" Gateway setting their `Accepted` condition to `False`. +Logically, this is equivalent to a single Gateway as below: -This MUST happen in the following scenarios: -- Attaching to a non-existing Gateway - -If a Gateway's `Accepted` condition is `False` then the conditions 'Programmed' and 'Ready' (if supported) MUST be set to `False`. +```yaml +apiVersion: gateway.networking.k8s.io/v1beta1 +kind: Gateway +metadata: + name: gateway +spec: + gatewayClassName: example + listeners: + - name: common-monitoring + port: 8080 + protocol: HTTP + - name: domain-a + hostname: a.example.com + protocol: HTTP + port: 80 +``` -Invalid Gateways MUST be rejected by the implementation during creation or update. This MUST happen in the following scenarios: -- Attaching to another "child" Gateway -- Attaching to oneself (self-referential) -- Attaching to a Gateway with a different `GatewayClassName` +### Semantics #### Route Attaching @@ -102,11 +212,37 @@ spec: sectionName: metrics ``` +Routes can only bind to listeners *directly* defined in the `Gateway` referenced. +For instance, the following route referencing a listener defined in the parent `Gateway`, but attaching to the child `Gateway` is not valid. +This will be reported in [status](#status-fields). + +```yaml +apiVersion: gateway.networking.k8s.io/v1beta1 +kind: HTTPRoute +metadata: + name: httproute-example +spec: + parentRefs: + - name: child-gateway + sectionName: section-in-parent +``` + #### Merging Spec Fields -##### GatewayClassName +##### Validation + +A parent and child MUST have the same `gatewayClassName`. +This will be detected by the implementation and reported in [status](#status-fields). + +A child resource MUST not set any `spec.infrastructure` fields beyond `attachTo`, and cannot set `spec.address`. +This can be validated in the CRD schema. + +A parent resource MUST not set `spec.infrastructure.attachTo`. +That is, we do not allow multiple tiers of Gateways chaining to each other; there is only a single parent with children.. +This can be validated in the CRD schema. -The `GatewayClassName` MUST be set on "child" and "parent" Gateways and be equivalent. +A child resource cannot `attachTo` any resource that doesn't allow attachment (i.e. it does not specify `spec.infrastructure.allowedChildren` for `Gateway`s). +This will be detected by the implementation and reported in [status](#status-fields). ##### Listeners @@ -121,11 +257,15 @@ kind: Gateway metadata: name: parent-gateway spec: + infrastructure: + allowedChildren: + namespaces: + from: Same gatewayClassName: example listeners: - name: HTTP port: 80 - protocol: HTTP + protocol: HTTP --- apiVersion: gateway.networking.k8s.io/v1beta1 kind: Gateway @@ -141,50 +281,62 @@ spec: listeners: - name: metrics port: 8080 - protocol: HTTP + protocol: HTTP ``` -With this configuration, the realized "primary" Gateway should listen on port `80` and `8080`. +With this configuration, the realized "parent" Gateway should listen on port `80` and `8080`. -###### Listener Precendence +###### Listener Precedence Gateway Listeners should be merged using the following precedence: - "parent" Gateway - "child" Gateway ordered by creation time (oldest first) - "child" Gateway ordered alphabetical by “{namespace}/{name}”. -###### Listener Conflict - -A "child" Gateway's listener status MUST set the condition `Conflicted` to `True` if it unable to merge with a "primary" Gateway. - -##### Addresses - -Merging `Addresses` is not supported and the "child" Gateway's `Addresses` MUST be empty. - +If there are conflicts between these, this should be reported as `Conflicted=True` in the listener as usual. ### Status Fields - #### Addresses The list of `Addresses` that appear in the status of the "child" Gateway MUST be the same as the "parent" Gateway. -#### Listeners +#### Gateway Conditions + +Gateway conditions currently supports the following condition types: `Accepted` and `Programmed` -The "child" Gateways MUST provide a status for each Listener defined in their specification. Listeners defined in "sibling" and the "parent" Gateway SHOULD NOT appear on a "child" Gateway. -Listeners in "child" Gateways SHOULD NOT appear on the "parent" Gateway. +For parent gateways, `Accepted` should be set based on the entire set of merged listeners. +For instance, if a child listener is invalid, `ListenersNotValid` would be reported. +`Programmed` is not expected, generally, to depend on the children resources, but if an implementation does depend on these +they should consider child resources when reporting this status. -#### Conditions +For child gateways, `Accepted` and `Programmed` should consider the overall merged Gateway status, but only the child's own listeners. -"Child" Gateways MUST support the same types of status conditions as "parent" Gateways. +For example, if I have a `parent`, `child-1`, and `child-2`: +* If parent is entirely invalid (for example, an invalid `address`), all three Gateways will reported `Accepted=False`. +* If `child-1` has an invalid listener, `parent` and `child-1` will report `ListenersNotValid`, while `child-2` will not. +* If `child-1` references a parent that doesn't exist then `child-1` will report `Accepted=False` +* If `child-1` references a parent that doesn't allow merging then `child-1` will report `Accepted=False` +* If `child-1` references another child (eg. `child-2`) then `child-1` will report `Accepted=False` +* If `child-1` references itself then `child-1` will report `Accepted=False` +* If `child-1` and `parent` have different gatewayClassNames then `child-1` will report `Accepted=False` +When reporting status of a child, an implementation SHOULD be cautious about what information from the parent or siblings are reported +to avoid accidentally leaking sensitive information that the child would not otherwise have access to. -## Future Goals +#### Listener Conditions -### Requirement Level +Listener conditions should only be set for listeners directly defined in a given Gateway. +Parent gateways MUST NOT have children's resources in their listener conditions list. +Children gateways MUST NOT have parent's or sibling's resources in their listener conditions list. -We want to keep this API very simple so that the merging requirement level could increase from `MAY` to `MUST` +#### Policy attachment +Policies attached to a parent Gateway apply to both the parent and all children listeners. + +Policies that attach to a child Gateway apply to all listeners defined in that Gateway, but do not impact +parent or sibling listeners. +If the implementation cannot apply the policy to only specific listeners, it should reject the policy. ## Alternatives @@ -210,11 +362,10 @@ status: ... Use of a label (ie. `gateway.networking.k8s.io/parent-gateway: name`) could be used to select child gateways vs using `spec.infrastructure.attachTo` ## References + Mentioned in Prior GEPs: - https://github.com/kubernetes-sigs/gateway-api/pull/1757 -Prior Discussions: +Prior Discussions: - https://github.com/kubernetes-sigs/gateway-api/discussions/1248 - https://github.com/kubernetes-sigs/gateway-api/discussions/1246 - - diff --git a/geps/gep-1713/metadata.yaml b/geps/gep-1713/metadata.yaml index 752beaba50..ff172a9ff8 100644 --- a/geps/gep-1713/metadata.yaml +++ b/geps/gep-1713/metadata.yaml @@ -5,3 +5,4 @@ name: Gateway Merging status: Provisional authors: - dprotaso + - howardjohn From e9148960fd2549922623e571c6de13c06b015025 Mon Sep 17 00:00:00 2001 From: dprotaso Date: Tue, 6 Feb 2024 10:25:09 -0500 Subject: [PATCH 16/36] undo removing future goal of making merging a MUST requirement --- geps/gep-1713/index.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/geps/gep-1713/index.md b/geps/gep-1713/index.md index b9d8f8cc63..3f94d4f5f9 100644 --- a/geps/gep-1713/index.md +++ b/geps/gep-1713/index.md @@ -190,7 +190,7 @@ spec: protocol: HTTP - name: domain-a hostname: a.example.com - protocol: HTTP + protocol: HTTPWe want to keep this API very simple so that the merging requirement level could increase from `MAY` to `MUST` port: 80 ``` @@ -338,6 +338,12 @@ Policies that attach to a child Gateway apply to all listeners defined in that G parent or sibling listeners. If the implementation cannot apply the policy to only specific listeners, it should reject the policy. +## Future Goals + +### Requirement Level + +We want to keep this API very simple so that the merging requirement level could increase from `MAY` to `MUST` + ## Alternatives #### New Resource From b3f7113067f9fc8148af6564f96d31c216befc41 Mon Sep 17 00:00:00 2001 From: dprotaso Date: Wed, 7 Feb 2024 12:25:38 -0500 Subject: [PATCH 17/36] fix some typos --- geps/gep-1713/index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/geps/gep-1713/index.md b/geps/gep-1713/index.md index 3f94d4f5f9..e128a740ea 100644 --- a/geps/gep-1713/index.md +++ b/geps/gep-1713/index.md @@ -190,7 +190,7 @@ spec: protocol: HTTP - name: domain-a hostname: a.example.com - protocol: HTTPWe want to keep this API very simple so that the merging requirement level could increase from `MAY` to `MUST` + protocol: HTTP port: 80 ``` @@ -238,7 +238,7 @@ A child resource MUST not set any `spec.infrastructure` fields beyond `attachTo` This can be validated in the CRD schema. A parent resource MUST not set `spec.infrastructure.attachTo`. -That is, we do not allow multiple tiers of Gateways chaining to each other; there is only a single parent with children.. +That is, we do not allow multiple tiers of Gateways chaining to each other; there is only a single parent with children. This can be validated in the CRD schema. A child resource cannot `attachTo` any resource that doesn't allow attachment (i.e. it does not specify `spec.infrastructure.allowedChildren` for `Gateway`s). From e094dff44895655d8f514cd550ee1484b9693162 Mon Sep 17 00:00:00 2001 From: dprotaso Date: Wed, 7 Feb 2024 13:09:18 -0500 Subject: [PATCH 18/36] attaching to a Gateway resource must be permitted --- geps/gep-1713/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/geps/gep-1713/index.md b/geps/gep-1713/index.md index e128a740ea..374cda3e34 100644 --- a/geps/gep-1713/index.md +++ b/geps/gep-1713/index.md @@ -241,7 +241,7 @@ A parent resource MUST not set `spec.infrastructure.attachTo`. That is, we do not allow multiple tiers of Gateways chaining to each other; there is only a single parent with children. This can be validated in the CRD schema. -A child resource cannot `attachTo` any resource that doesn't allow attachment (i.e. it does not specify `spec.infrastructure.allowedChildren` for `Gateway`s). +A child resource cannot `attachTo` any Gateway resource that doesn't allow attachment (eg. it does not specify `spec.infrastructure.allowedChildren` for `Gateway`s). This will be detected by the implementation and reported in [status](#status-fields). ##### Listeners From dcdd14214c745a5206723d814bab4c649899f3f4 Mon Sep 17 00:00:00 2001 From: dprotaso Date: Mon, 12 Feb 2024 17:47:10 -0500 Subject: [PATCH 19/36] allow empty sectionName/port when targetting a child Gateway --- geps/gep-1713/index.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/geps/gep-1713/index.md b/geps/gep-1713/index.md index 374cda3e34..adae3b1a9a 100644 --- a/geps/gep-1713/index.md +++ b/geps/gep-1713/index.md @@ -139,7 +139,7 @@ Below shows an example of an end to end configuration with Gateway merging. Here we define a parent resource, which allows children from the same namespace. A single listener is defined in the parent. -The child gateway attaches to this Gateway and specifies an additional listener. +The child Gateway attaches to this Gateway and specifies an additional listener. ```yaml apiVersion: gateway.networking.k8s.io/v1beta1 @@ -198,8 +198,9 @@ spec: #### Route Attaching -Routes MUST be able to specify a "child" Gateways as a `parentRef` and make use of the fields in `ParentReference` to help target a specific listener. - +Routes MUST be able to specify a child Gateway as a `parentRef` and make use of the fields in `ParentReference` to help target a specific listener. +If no listener is targeted (`sectionName`/`port` are unset) then the Route references all the listeners on the child Gateway. It MUST NOT attach +to a listener on a parent or sibling Gateway. ```yaml apiVersion: gateway.networking.k8s.io/v1beta1 From 668af46c009b7b678aac5a55065efb2b26922772 Mon Sep 17 00:00:00 2001 From: dprotaso Date: Mon, 12 Feb 2024 17:50:44 -0500 Subject: [PATCH 20/36] Add TooManyListeners failure mode --- geps/gep-1713/index.md | 1 + 1 file changed, 1 insertion(+) diff --git a/geps/gep-1713/index.md b/geps/gep-1713/index.md index adae3b1a9a..f583008589 100644 --- a/geps/gep-1713/index.md +++ b/geps/gep-1713/index.md @@ -330,6 +330,7 @@ to avoid accidentally leaking sensitive information that the child would not oth Listener conditions should only be set for listeners directly defined in a given Gateway. Parent gateways MUST NOT have children's resources in their listener conditions list. Children gateways MUST NOT have parent's or sibling's resources in their listener conditions list. +An implementation MAY reject Listeners with ListenerConditionAccepted=False and Reason "TooManyListeners" #### Policy attachment From ec9ce6377f883c596e69faaf30fcde7e4784591e Mon Sep 17 00:00:00 2001 From: dprotaso Date: Mon, 12 Feb 2024 17:57:05 -0500 Subject: [PATCH 21/36] address feedback from sunjay --- geps/gep-1713/index.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/geps/gep-1713/index.md b/geps/gep-1713/index.md index f583008589..20c1a78709 100644 --- a/geps/gep-1713/index.md +++ b/geps/gep-1713/index.md @@ -311,16 +311,18 @@ For instance, if a child listener is invalid, `ListenersNotValid` would be repor `Programmed` is not expected, generally, to depend on the children resources, but if an implementation does depend on these they should consider child resources when reporting this status. -For child gateways, `Accepted` and `Programmed` should consider the overall merged Gateway status, but only the child's own listeners. +For child gateways, `Accepted` and `Programmed` should not consider the overall merged Gateway status, but only the child's own listeners. +The exception to this is if the parent Gateway is not valid in any regard. For example, if I have a `parent`, `child-1`, and `child-2`: -* If parent is entirely invalid (for example, an invalid `address`), all three Gateways will reported `Accepted=False`. +* If `parent` is entirely invalid (for example, an invalid `address`), all three Gateways will reported `Accepted=False`. * If `child-1` has an invalid listener, `parent` and `child-1` will report `ListenersNotValid`, while `child-2` will not. * If `child-1` references a parent that doesn't exist then `child-1` will report `Accepted=False` * If `child-1` references a parent that doesn't allow merging then `child-1` will report `Accepted=False` * If `child-1` references another child (eg. `child-2`) then `child-1` will report `Accepted=False` * If `child-1` references itself then `child-1` will report `Accepted=False` * If `child-1` and `parent` have different gatewayClassNames then `child-1` will report `Accepted=False` +* If `child-1` references a parent whose allowed child namespaces do not include `child-1`'s namespace, then `child-1` will report `Accepted=False` When reporting status of a child, an implementation SHOULD be cautious about what information from the parent or siblings are reported to avoid accidentally leaking sensitive information that the child would not otherwise have access to. From d3ca6b17ba8bd59ce7727efaf8c05c1508b0334e Mon Sep 17 00:00:00 2001 From: Dave Protasowski Date: Fri, 15 Mar 2024 11:01:32 -0400 Subject: [PATCH 22/36] Update geps/gep-1713/index.md Co-authored-by: Rob Scott --- geps/gep-1713/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/geps/gep-1713/index.md b/geps/gep-1713/index.md index 20c1a78709..1ee09575f6 100644 --- a/geps/gep-1713/index.md +++ b/geps/gep-1713/index.md @@ -25,7 +25,7 @@ There can be O(1000) Knative Services in the cluster which means we have O(1000) Thus updating a single Gateway resource with this many certificates is a contention point and inhibits horizontal scaling of our controllers. [Istio Ambient](https://istio.io/v1.15/blog/2022/introducing-ambient-mesh/), similarly, creates a listener per Kubernetes service. -More broadly, large scale gateway users often expose O(1000) domains, but are currently limited by the maximum of 16 `listeners`. +More broadly, large scale gateway users often expose O(1000) domains, but are currently limited by the maximum of 64 `listeners`. The spec currently has language to indicate implementations `MAY` merge Gateways resources but the mechanic isn't defined. https://github.com/kubernetes-sigs/gateway-api/blob/541e9fc2b3c2f62915cb58dc0ee5e43e4096b3e2/apis/v1beta1/gateway_types.go#L76-L78 From 5d6853ddac05e2028452dbac50e641d2117fa4b1 Mon Sep 17 00:00:00 2001 From: Dave Protasowski Date: Fri, 15 Mar 2024 11:07:53 -0400 Subject: [PATCH 23/36] Update geps/gep-1713/index.md Co-authored-by: Rob Scott --- geps/gep-1713/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/geps/gep-1713/index.md b/geps/gep-1713/index.md index 1ee09575f6..4769812d26 100644 --- a/geps/gep-1713/index.md +++ b/geps/gep-1713/index.md @@ -111,7 +111,7 @@ type GatewayObjectReference struct { // When unspecified or empty string, core API group is inferred. // // +optional - // +kubebuilder:default="" + // +kubebuilder:default="gateway.networking.k8s.io" Group *Group `json:"group"` // Kind is kind of the referent. For example "Gateway". From aa2d64935a113c63a47918de736d93b615cd0d53 Mon Sep 17 00:00:00 2001 From: Dave Protasowski Date: Fri, 15 Mar 2024 11:55:15 -0400 Subject: [PATCH 24/36] Update geps/gep-1713/index.md Co-authored-by: Rob Scott --- geps/gep-1713/index.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/geps/gep-1713/index.md b/geps/gep-1713/index.md index 4769812d26..66ccbc2c49 100644 --- a/geps/gep-1713/index.md +++ b/geps/gep-1713/index.md @@ -107,8 +107,7 @@ type ChildrenNamespaces struct { // GatewayObjectReference identifies an API object including its namespace, // defaulting to Gateway. type GatewayObjectReference struct { - // Group is the group of the referent. For example, "gateway.networking.k8s.io". - // When unspecified or empty string, core API group is inferred. + // Group is the group of the referent. // // +optional // +kubebuilder:default="gateway.networking.k8s.io" From df3b0dc0883e5e466a9253d880916fea75552fa1 Mon Sep 17 00:00:00 2001 From: Dave Protasowski Date: Wed, 3 Jul 2024 12:19:29 -0400 Subject: [PATCH 25/36] rename GatewayObjectReference to ParentGatewayReference --- geps/gep-1713/index.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/geps/gep-1713/index.md b/geps/gep-1713/index.md index 66ccbc2c49..9a13fd2308 100644 --- a/geps/gep-1713/index.md +++ b/geps/gep-1713/index.md @@ -40,8 +40,8 @@ If, and only if, they have done so, the same Gateway is also permitted to leave A "child" Gateway references a parent Gateway. This is done with a new field `attachTo` in the `spec.infrastructure` stanza of a Gateway. -The `attachTo` field is a new type `GatewayObjectReference`. -Although the use of `GatewayObjectReference` allows users to attach to any `kind`, this GEP only defines the behavior of attaching a Gateway to another Gateway. +The `attachTo` field is a new type `GatewayParentReference`. +Although the use of `GatewayParentReference` allows users to attach to any `kind`, this GEP only defines the behavior of attaching a Gateway to another Gateway. A "sibling" is a Gateway that shares a parent with another child Gateway. @@ -63,7 +63,7 @@ type GatewayInfrastructure struct { // AttachTo allows the Gateway to associate itself with another resource. // A common scenario is to reference another Gateway which marks // this Gateway a child of another. - AttachTo GatewayObjectReference `json:"attachTo"` + AttachTo GatewayParentReference `json:"attachTo"` } // AllowedChildren defines which objects may be attached as children @@ -104,9 +104,9 @@ type ChildrenNamespaces struct { Selector *metav1.LabelSelector `json:"selector,omitempty"` } -// GatewayObjectReference identifies an API object including its namespace, +// ParentGatewayReference identifies an API object including its namespace, // defaulting to Gateway. -type GatewayObjectReference struct { +type ParentGatewayReference struct { // Group is the group of the referent. // // +optional From 6e4e7ab50fde69e5e67c68a900bd8be091ec7a74 Mon Sep 17 00:00:00 2001 From: Dave Protasowski Date: Wed, 3 Jul 2024 12:35:42 -0400 Subject: [PATCH 26/36] Include a count of attached child gateways --- geps/gep-1713/index.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/geps/gep-1713/index.md b/geps/gep-1713/index.md index 9a13fd2308..8278ce58ab 100644 --- a/geps/gep-1713/index.md +++ b/geps/gep-1713/index.md @@ -301,6 +301,10 @@ If there are conflicts between these, this should be reported as `Conflicted=Tru The list of `Addresses` that appear in the status of the "child" Gateway MUST be the same as the "parent" Gateway. +#### AttachedGateways + +The property `attachedGateways` will appear on the status of the "parent" Gateway will indicate the total number of "child" Gateways that have are attached + #### Gateway Conditions Gateway conditions currently supports the following condition types: `Accepted` and `Programmed` @@ -333,7 +337,7 @@ Parent gateways MUST NOT have children's resources in their listener conditions Children gateways MUST NOT have parent's or sibling's resources in their listener conditions list. An implementation MAY reject Listeners with ListenerConditionAccepted=False and Reason "TooManyListeners" -#### Policy attachment +#### Policy Attachment Policies attached to a parent Gateway apply to both the parent and all children listeners. From 23f5e157c859608a325bfc743fdbd9b500cf66a0 Mon Sep 17 00:00:00 2001 From: Dave Protasowski Date: Wed, 3 Jul 2024 14:43:47 -0400 Subject: [PATCH 27/36] Include some more alternative solutions --- geps/gep-1713/index.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/geps/gep-1713/index.md b/geps/gep-1713/index.md index 8278ce58ab..ad3a41e0bb 100644 --- a/geps/gep-1713/index.md +++ b/geps/gep-1713/index.md @@ -382,3 +382,16 @@ Mentioned in Prior GEPs: Prior Discussions: - https://github.com/kubernetes-sigs/gateway-api/discussions/1248 - https://github.com/kubernetes-sigs/gateway-api/discussions/1246 + +#### Use of Multiple Disjointed Gateways + +An alternative would be to encourage users to not use overly large Gateways to minimize the blast radius of any issues. Use of disjoint Gateways could accomplish this but it has the disadvantage of consuming more resources and introducing complexity when it comes to operations work (eg. setting up DNS records etc.) + +#### Increase the Listener Limit + +Increasing the limit may help in situations where you are creating many listeners such as adding certificates created using an ACME HTTP01 challenge. Unfortunately this still makes the Gateway a single point of contention. Unfortunately, there will always be an upper bound because of etcd limitations. +For workloads like Knative we can have O(1000) Services on the cluster with unique subdomains. + +#### Expand Route Functionality + +For workloads with many certificates one option would be to introduce a `tls` stanza somewhere in the Route types. These Routes would then attach to a single Gateway. Then application operators can provide their own certificates. This probably would require some ability to have a handshake agreement with the Gateway. From 969c5a11309b86ff369e8c64d6dbb56ce266e163 Mon Sep 17 00:00:00 2001 From: Dave Protasowski Date: Wed, 3 Jul 2024 14:46:56 -0400 Subject: [PATCH 28/36] mention route delegation GEP --- geps/gep-1713/index.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/geps/gep-1713/index.md b/geps/gep-1713/index.md index ad3a41e0bb..9eff181b43 100644 --- a/geps/gep-1713/index.md +++ b/geps/gep-1713/index.md @@ -395,3 +395,5 @@ For workloads like Knative we can have O(1000) Services on the cluster with uniq #### Expand Route Functionality For workloads with many certificates one option would be to introduce a `tls` stanza somewhere in the Route types. These Routes would then attach to a single Gateway. Then application operators can provide their own certificates. This probably would require some ability to have a handshake agreement with the Gateway. + +Sorta related there was a Route Delegation GEP (https://github.com/kubernetes-sigs/gateway-api/issues/1058) that was abandoned From 4e05526bb0affe51d584c65024f8311ebd6f041c Mon Sep 17 00:00:00 2001 From: Dave Protasowski Date: Wed, 3 Jul 2024 15:04:46 -0400 Subject: [PATCH 29/36] mention use of CEL --- geps/gep-1713/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/geps/gep-1713/index.md b/geps/gep-1713/index.md index 9eff181b43..1390ed0f05 100644 --- a/geps/gep-1713/index.md +++ b/geps/gep-1713/index.md @@ -235,7 +235,7 @@ A parent and child MUST have the same `gatewayClassName`. This will be detected by the implementation and reported in [status](#status-fields). A child resource MUST not set any `spec.infrastructure` fields beyond `attachTo`, and cannot set `spec.address`. -This can be validated in the CRD schema. +This can be validated in the CRD schema potentially using CEL. A parent resource MUST not set `spec.infrastructure.attachTo`. That is, we do not allow multiple tiers of Gateways chaining to each other; there is only a single parent with children. From a979bca19f4588f01a2cbf49627eae10a6634d1f Mon Sep 17 00:00:00 2001 From: Dave Protasowski Date: Wed, 3 Jul 2024 15:28:22 -0400 Subject: [PATCH 30/36] fix references section --- geps/gep-1713/index.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/geps/gep-1713/index.md b/geps/gep-1713/index.md index 1390ed0f05..1a1bab0bc8 100644 --- a/geps/gep-1713/index.md +++ b/geps/gep-1713/index.md @@ -374,15 +374,6 @@ status: ... Use of a label (ie. `gateway.networking.k8s.io/parent-gateway: name`) could be used to select child gateways vs using `spec.infrastructure.attachTo` -## References - -Mentioned in Prior GEPs: -- https://github.com/kubernetes-sigs/gateway-api/pull/1757 - -Prior Discussions: -- https://github.com/kubernetes-sigs/gateway-api/discussions/1248 -- https://github.com/kubernetes-sigs/gateway-api/discussions/1246 - #### Use of Multiple Disjointed Gateways An alternative would be to encourage users to not use overly large Gateways to minimize the blast radius of any issues. Use of disjoint Gateways could accomplish this but it has the disadvantage of consuming more resources and introducing complexity when it comes to operations work (eg. setting up DNS records etc.) @@ -397,3 +388,12 @@ For workloads like Knative we can have O(1000) Services on the cluster with uniq For workloads with many certificates one option would be to introduce a `tls` stanza somewhere in the Route types. These Routes would then attach to a single Gateway. Then application operators can provide their own certificates. This probably would require some ability to have a handshake agreement with the Gateway. Sorta related there was a Route Delegation GEP (https://github.com/kubernetes-sigs/gateway-api/issues/1058) that was abandoned + +## References + +Mentioned in Prior GEPs: +- https://github.com/kubernetes-sigs/gateway-api/pull/1757 + +Prior Discussions: +- https://github.com/kubernetes-sigs/gateway-api/discussions/1248 +- https://github.com/kubernetes-sigs/gateway-api/discussions/1246 From 98eca501fddc39b1e714139e148b18432253e3f5 Mon Sep 17 00:00:00 2001 From: Dave Protasowski Date: Wed, 3 Jul 2024 15:44:21 -0400 Subject: [PATCH 31/36] include comments why some alternatives were rejected --- geps/gep-1713/index.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/geps/gep-1713/index.md b/geps/gep-1713/index.md index 1a1bab0bc8..4bded870ed 100644 --- a/geps/gep-1713/index.md +++ b/geps/gep-1713/index.md @@ -370,9 +370,17 @@ spec: status: ... ``` +This idea was rejected because + +From youngnick in https://github.com/kubernetes-sigs/gateway-api/discussions/1248#discussioncomment-3097235 + +> I'm a big -1 on having a separate Listeners object, because it will need the full panoply of the two-way handshake between the Gateway and Listener objects, just like the ones between Gateway and Listener, because you know that as soon as you separate them, people will want them in different namespaces. And that's on top of the ETOOMANYCRDs problem. + #### Use of the `gateway.networking.k8s.io/parent-gateway` label -Use of a label (ie. `gateway.networking.k8s.io/parent-gateway: name`) could be used to select child gateways vs using `spec.infrastructure.attachTo` +Use of a label (ie. `gateway.networking.k8s.io/parent-gateway: name`) could be used to select child gateways instead using `spec.infrastructure.attachTo` + +Use of labels is discouraged since it doesn't provide any extensibility. #### Use of Multiple Disjointed Gateways From 0de79b6435c843d6e516714eabcdca40bb26ba3c Mon Sep 17 00:00:00 2001 From: Dave Protasowski Date: Wed, 3 Jul 2024 16:44:23 -0400 Subject: [PATCH 32/36] include a case where two child gateways conflict --- geps/gep-1713/index.md | 1 + 1 file changed, 1 insertion(+) diff --git a/geps/gep-1713/index.md b/geps/gep-1713/index.md index 4bded870ed..61992d6305 100644 --- a/geps/gep-1713/index.md +++ b/geps/gep-1713/index.md @@ -326,6 +326,7 @@ For example, if I have a `parent`, `child-1`, and `child-2`: * If `child-1` references itself then `child-1` will report `Accepted=False` * If `child-1` and `parent` have different gatewayClassNames then `child-1` will report `Accepted=False` * If `child-1` references a parent whose allowed child namespaces do not include `child-1`'s namespace, then `child-1` will report `Accepted=False` +* If `child-1` is valid, then when `child-2` is created if it conflicts with `child-1` then `child-2` will report `Accepted=False`. `child-1` status conditions will remain unchanged. `parent` will report `ListenersNotValid` When reporting status of a child, an implementation SHOULD be cautious about what information from the parent or siblings are reported to avoid accidentally leaking sensitive information that the child would not otherwise have access to. From 9add2e6709870ae342416da124d47f3f952843a0 Mon Sep 17 00:00:00 2001 From: Dave Protasowski Date: Wed, 3 Jul 2024 17:33:15 -0400 Subject: [PATCH 33/36] change word to mechanism --- geps/gep-1713/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/geps/gep-1713/index.md b/geps/gep-1713/index.md index 61992d6305..2e0dde66db 100644 --- a/geps/gep-1713/index.md +++ b/geps/gep-1713/index.md @@ -11,7 +11,7 @@ The Gateway Resource is a contention point since it is the only place to attach ## Goals -- Define a mechanic to merge multiple Gateways (logically) +- Define a mechanism to merge multiple Gateways (logically) - Define a set of acceptable properties that can be merged and their semantics ## Non-Goals From 01cb1fdc167cae681c72f84bbc36614044d4bd5c Mon Sep 17 00:00:00 2001 From: Dave Protasowski Date: Wed, 3 Jul 2024 19:48:00 -0400 Subject: [PATCH 34/36] Update geps/gep-1713/index.md Co-authored-by: Candace Holman --- geps/gep-1713/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/geps/gep-1713/index.md b/geps/gep-1713/index.md index 2e0dde66db..738c73633b 100644 --- a/geps/gep-1713/index.md +++ b/geps/gep-1713/index.md @@ -303,7 +303,7 @@ The list of `Addresses` that appear in the status of the "child" Gateway MUST be #### AttachedGateways -The property `attachedGateways` will appear on the status of the "parent" Gateway will indicate the total number of "child" Gateways that have are attached +The property `attachedGateways` will appear on the status of the "parent" Gateway and will indicate the total number of "child" Gateways that are attached. #### Gateway Conditions From df249a5dd89ce94cb19a0cde7fd701c34b1bc55a Mon Sep 17 00:00:00 2001 From: Dave Protasowski Date: Wed, 3 Jul 2024 19:56:27 -0400 Subject: [PATCH 35/36] add numbers to order precedence --- geps/gep-1713/index.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/geps/gep-1713/index.md b/geps/gep-1713/index.md index 738c73633b..cad408100a 100644 --- a/geps/gep-1713/index.md +++ b/geps/gep-1713/index.md @@ -289,9 +289,9 @@ With this configuration, the realized "parent" Gateway should listen on port `80 ###### Listener Precedence Gateway Listeners should be merged using the following precedence: -- "parent" Gateway -- "child" Gateway ordered by creation time (oldest first) -- "child" Gateway ordered alphabetical by “{namespace}/{name}”. +1. "parent" Gateway +2. "child" Gateway ordered by creation time (oldest first) +3. "child" Gateway ordered alphabetical by “{namespace}/{name}”. If there are conflicts between these, this should be reported as `Conflicted=True` in the listener as usual. From 43074ed0bdcad98ec0aa7c04f93d71e955ea7847 Mon Sep 17 00:00:00 2001 From: Dave Protasowski Date: Wed, 3 Jul 2024 20:05:12 -0400 Subject: [PATCH 36/36] Update language about gateway conflict and direct them to the gateway status condition section --- geps/gep-1713/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/geps/gep-1713/index.md b/geps/gep-1713/index.md index cad408100a..025dc6fbb3 100644 --- a/geps/gep-1713/index.md +++ b/geps/gep-1713/index.md @@ -293,7 +293,7 @@ Gateway Listeners should be merged using the following precedence: 2. "child" Gateway ordered by creation time (oldest first) 3. "child" Gateway ordered alphabetical by “{namespace}/{name}”. -If there are conflicts between these, this should be reported as `Conflicted=True` in the listener as usual. +If there are listener conflicts between Gateways, this should be reported as `Conflicted=True` in the ListenerStatus as usual. See 'Gateway Conditions' section below for more details on which Gateway resource should report the conflict. ### Status Fields