GEP-1867: Per-Gateway Infrastructure

geps/gep-1867.md

@@ -0,0 +1,128 @@
+# GEP-1867: Per-Gateway Infrastructure
+
+* Status: Provisional
+* Issue: [#1876](https://github.com/kubernetes-sigs/gateway-api/issues/1876)
+
+## Overview
+
+`Gateway`s represent a piece of infrastructure implemented by cloud load balancers, in-cluster deployments, or other mechanisms.
+These often need vendor-specific configuration outside the scope of existing APIs (e.g. "size" or "version" of the infrastructure to provision).
+
+Today `GatewayClass.spec.parametersRef` is available to attach arbitrary configuration to a `GatewayClass`.
+
+This GEP will explain why that is not sufficient to meet common use cases, and introduce a new field - `infrastructure` - to address these cases.
+
+Related discussions:
+* [Support cluster-local Gateways](https://github.com/kubernetes-sigs/gateway-api/discussions/1247)
+* [Scaling Gateway Resources](https://github.com/kubernetes-sigs/gateway-api/discussions/1355)
+* [Manual deployments](https://github.com/kubernetes-sigs/gateway-api/issues/1687)
+* [Merging Gateways](https://github.com/kubernetes-sigs/gateway-api/pull/1863)
+* [In Cluster Gateway Deployments](https://github.com/kubernetes-sigs/gateway-api/pull/1757)
+
+## Goals
+
+* Provide the ability to configure arbitrary (implementation specific) attributes about a **specific Gateway**.
+* Provide the ability to configure a standardized set of attributes about a **specific Gateway**.
+
+## Challenges with GatewayClass
+
+`GatewayClass.spec.parametersRef` is the existing mechanism to configure arbitrary fields on a Gateway.
+However, this introduces operational challenges when configuring Gateways.
+
+### Scope
+
+As a `Gateway` manager (with RBAC permissions to a specific `Gateway`) I should be able to declaratively make changes to that `Gateway` without the need for access to cluster-scoped resources (`GatewayClass`) and without affecting other `Gateways` managed by the same `GatewayClass`.
+This has been previously discussed in [this issue](https://github.com/kubernetes-sigs/gateway-api/issues/567).
+
+As a cluster scoped resource, `GatewayClass` does not meet this requirement.
+This restricts customization use cases to either a few pre-provisioned classes by the admin, or running in an environment where the "Infrastructure Provider" and "Cluster Operator" are the same roles.
+The distinction between these roles is explicitly called out on the homepage: https://gateway-api.sigs.k8s.io/#what-is-the-gateway-api.
+
+### Custom Resource
+
+`parametersRef` is entirely a generic implementation-specific meaning.
+This means implementations will either need a custom CRD or use untyped resources like ConfigMap.
+Neither of these have any consistency between implementations.
+While there will always be some vendor-specific requirements, there are also a number of configuration aspects of a Gateway that are common between implementations.
+However, these cannot currently be expressed in a vendor-neutral way.
+
+Additionally, there is hesitancy to use a CRD (which leads to CRD proliferation), which pushes users towards untyped ConfigMaps which are not much better than annotations. The scoping, as mentioned above, is also a bit awkward of a cluster scoped resource pointing to a namespaced object.
+
+### Separation of concerns
+
+While there is value out of providing class-wide options as defaults, there is also value in providing these options on the object (Gateway) directly.
+
+Some parallels in existing APIs:
+
+[Policy Attachment](https://gateway-api.sigs.k8s.io/references/policy-attachment) offers a hierarchy of defaults and overrides, allowing attachment to GatewayClass and Gateway.
+This is similar to our needs here, but representing infrastructure configuration as a "Policy" is a bit problematic, and the existing mechanisms have no hierarchy.
+
+In core Kubernetes, Pods declare their requirements (for example, CPU requests) inline in the Pod resource; there is not a `ResourceClass` API that abstracts these further.
+These higher level abstractions are handled by layered APIs (whether this is a CRD, an admission webhook, CI/CD tooling, etc).
+This allows users the flexibility to easily configure things per-pod basis.
+If the infrastructure admin wants to impose defaults or requirements on this flexibility, they are able to do so (in fact, `LimitRanger` provides a built in mechanism to do so).
+
+### Dynamic Changes
+
+Currently, the spec recommends `GatewayClass` to be used as a *template*.
+Changes to it are not expected to change deployed `Gateway`s.
+
+This makes usage problematic in a declarative way.
+For example, if I wanted to represent a `version` field and change that to trigger an upgrade, I would need to create an entirely new `Gateway`.
+
+## API
+
+In order to address the concerns above, I propose a standard `infrastructure` API is added to `Gateway` and `GatewayClass`.
+Note the important part of this is the `Gateway` change; the `GatewayClass` aspect is mostly for consistency.
+
+Note that many of the fields under `GatewayInfrastructure` refer to work-in-progress GEPs.
+These fields are not tied to this GEP, and will only be included if the respective GEPs are approved.
+However, they are shown below to give a sense of the use cases the field can support.
+
+```go
+type GatewaySpec struct {
+  // Infrastructure defines infrastructure level attributes about this Gateway instance.
+  Infrastructure GatewayInfrastructure `json:"infrastructure"`
+  // ...
+}
+
+type GatewayClassSpec struct {
+  // Infrastructure defines infrastructure level attributes for all Gateways in this class.
+  // A Gateway may provide configuration for the same values; as all fields in GatewayInfrastructure are implementation specific,
+  // the merging logic between these is as well. However, the GatewayClass is generally expected to be providing defaults
+  // rather than overrides
+  Infrastructure GatewayInfrastructure `json:"infrastructure"`
+  // ...
+}
+
+type GatewayInfrastructure struct {
+  // AttachTo marks this Gateway as a child to the referenced parent Gateway.
+  // See GEP-1713 for details, which this depends on.
+  AttachTo LocalObjectReference
+
+  // Scope defines the "scope" of the Gateway's addresses.
+  // Valid options are ClusterLocal or Public
+  // See GEP-1651 for details, which this depends on.
+  // NOTE: that GEP is likely to be implemented in a different way, at which point this field can be removed from this struct.
+  Scope AddressScope
+
+  // Version defines the version of the Gateway infrastructure to use.
+  // This is an opaque string that is implementation specific.
+  // Some possible valid examples include "canary", "v1.2.3", "docker.io/example:latest".
+  Version string
+
+  // Size defines the size of the Gateway infrastructure.
+  // The meaning of this is implementation specific.
+  // GEP-1762 will recommend this to mean "Pod Replicas" for in-cluster deployments.
+  Size int
+
+  // ParametersRef provides a arbitrary implementation-specific configuration for
+  // fields not expressed directly in this struct.
+  // This follows the same semantics as GatewayClass's ParametersRef, but lives on the Gateway.
+  ParametersRef ParametersReference
+}
+```
+
+The exact fields contained in `GatewayInfrastructure` are subject to change, and additional common fields can be added as use cases arise.
+However, this will suffer from the common problem of a tension between providing common fields against offering extension points, so we should ensure that we are cautious to prevent excessive bloating of the field.
+Fields in `GatewayInfrastructure` should be attributes of infrastructure that are relatively common across implementations.