Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update minimum k8s version to 1.25 #1885

Merged
merged 3 commits into from
Apr 26, 2024
Merged

Update minimum k8s version to 1.25 #1885

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
bbba1c6
Update minimum k8s version to 1.25
lucacome Apr 25, 2024
ae36e63
Remove webook from tests
lucacome Apr 26, 2024
90f8116
Merge branch 'main' into chore/remove-webhook
lucacome Apr 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ jobs:
id: vars
run: |
echo "go_path=$(go env GOPATH)" >> $GITHUB_OUTPUT
echo "min_k8s_version=1.23.17" >> $GITHUB_OUTPUT
echo "min_k8s_version=1.25.16" >> $GITHUB_OUTPUT

- name: Check if go.mod and go.sum are up to date
run: go mod tidy && git diff --exit-code -- go.mod go.sum
Expand Down
1 change: 0 additions & 1 deletion .github/workflows/conformance.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,6 @@ jobs:
ngf_prefix=ghcr.io/nginxinc/nginx-gateway-fabric
ngf_tag=${{ steps.ngf-meta.outputs.version }}
if [ ${{ github.event_name }} == "schedule" ]; then export GW_API_VERSION=main; fi
if [ ${{ startsWith(inputs.k8s-version, '1.23') || startsWith(inputs.k8s-version, '1.24') }} == "true" ]; then export INSTALL_WEBHOOK=true; fi
if [ ${{ inputs.enable-experimental }} == "true" ]; then export ENABLE_EXPERIMENTAL=true; fi
make install-ngf-local-no-build${{ inputs.image == 'plus' && '-with-plus' || ''}} PREFIX=${ngf_prefix} TAG=${ngf_tag}
working-directory: ./conformance
Expand Down
5 changes: 2 additions & 3 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,6 @@ TELEMETRY_ENDPOINT=# if empty, NGF will report telemetry in its logs at debug le
TELEMETRY_ENDPOINT_INSECURE = false

GW_API_VERSION = 1.0.0
INSTALL_WEBHOOK = false
NODE_VERSION = $(shell cat .nvmrc)

# go build flags - should not be overridden by the user
Expand Down Expand Up @@ -193,12 +192,12 @@ install-ngf-local-build-with-plus: build-images-with-plus load-images-with-plus

.PHONY: helm-install-local
helm-install-local: ## Helm install NGF on configured kind cluster with local images. To build, load, and install with helm run make install-ngf-local-build.
./conformance/scripts/install-gateway.sh $(GW_API_VERSION) $(INSTALL_WEBHOOK)
./conformance/scripts/install-gateway.sh $(GW_API_VERSION)
helm install dev $(CHART_DIR) --create-namespace --wait --set service.type=NodePort --set nginxGateway.image.repository=$(PREFIX) --set nginxGateway.image.tag=$(TAG) --set nginxGateway.image.pullPolicy=Never --set nginx.image.repository=$(NGINX_PREFIX) --set nginx.image.tag=$(TAG) --set nginx.image.pullPolicy=Never -n nginx-gateway

.PHONY: helm-install-local-with-plus
helm-install-local-with-plus: ## Helm install NGF with NGINX Plus on configured kind cluster with local images. To build, load, and install with helm run make install-ngf-local-build-with-plus.
./conformance/scripts/install-gateway.sh $(GW_API_VERSION) $(INSTALL_WEBHOOK)
./conformance/scripts/install-gateway.sh $(GW_API_VERSION)
helm install dev $(CHART_DIR) --create-namespace --wait --set service.type=NodePort --set nginxGateway.image.repository=$(PREFIX) --set nginxGateway.image.tag=$(TAG) --set nginxGateway.image.pullPolicy=Never --set nginx.image.repository=$(NGINX_PLUS_PREFIX) --set nginx.image.tag=$(TAG) --set nginx.image.pullPolicy=Never --set nginx.plus=true -n nginx-gateway

# Debug Targets
Expand Down
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ The following table lists the software versions NGINX Gateway Fabric supports.

| NGINX Gateway Fabric | Gateway API | Kubernetes | NGINX OSS | NGINX Plus |
| -------------------- | ----------- | ---------- | --------- | ---------- |
| Edge | 1.0.0 | 1.23+ | 1.25.4 | R31 |
| Edge | 1.0.0 | 1.25+ | 1.25.5 | R31 |
| 1.2.0 | 1.0.0 | 1.23+ | 1.25.4 | R31 |
| 1.1.0 | 1.0.0 | 1.23+ | 1.25.3 | n/a |
| 1.0.0 | 0.8.1 | 1.23+ | 1.25.2 | n/a |
Expand Down
31 changes: 0 additions & 31 deletions charts/nginx-gateway-fabric/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,18 +43,6 @@ This chart deploys the NGINX Gateway Fabric in your Kubernetes cluster.
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/standard-install.yaml
```

If you are running on Kubernetes 1.23 or 1.24, you also need to install the validating webhook. To do so, run:

```shell
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/webhook-install.yaml
```

> **Important**
>
> The validating webhook is not needed if you are running Kubernetes 1.25+. Validation is done using CEL on the
> CRDs. See the [resource validation doc](https://docs.nginx.com/nginx-gateway-fabric/overview/resource-validation/)
> for more information.

## Installing the Chart

### Installing the Chart from the OCI Registry
Expand Down Expand Up @@ -143,19 +131,6 @@ To upgrade the Gateway CRDs from [the Gateway API repo](https://github.com/kuber
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/standard-install.yaml
```

If you are running on Kubernetes 1.23 or 1.24, you also need to update the validating webhook. To do so, run:

```shell
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/webhook-install.yaml
```

If you are running on Kubernetes 1.25 or newer and have the validating webhook installed, you should remove the
webhook. To do so, run:

```shell
kubectl delete -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/webhook-install.yaml
```

### Upgrading the CRDs

Helm does not upgrade the NGINX Gateway Fabric CRDs during a release upgrade. Before you upgrade a release, you
Expand Down Expand Up @@ -266,12 +241,6 @@ To delete the Gateway API CRDs from [the Gateway API repo](https://github.com/ku
kubectl delete -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/standard-install.yaml
```

If you are running on Kubernetes 1.23 or 1.24, you also need to delete the validating webhook. To do so, run:

```shell
kubectl delete -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/webhook-install.yaml
```

## Configuration

The following tables lists the configurable parameters of the NGINX Gateway Fabric chart and their default values.
Expand Down
7 changes: 3 additions & 4 deletions conformance/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,6 @@ NGF_MANIFEST=../deploy/manifests/nginx-gateway.yaml
CRDS=../deploy/manifests/crds/
STATIC_MANIFEST=provisioner/static-deployment.yaml
PROVISIONER_MANIFEST=provisioner/provisioner.yaml
INSTALL_WEBHOOK ?= false
ENABLE_EXPERIMENTAL ?= false
.DEFAULT_GOAL := help

Expand Down Expand Up @@ -62,13 +61,13 @@ load-images-with-plus: ## Load NGF and NGINX Plus images on configured kind clus

.PHONY: prepare-ngf-dependencies
prepare-ngf-dependencies: update-ngf-manifest ## Install NGF dependencies on configured kind cluster
./scripts/install-gateway.sh $(GW_API_VERSION) $(INSTALL_WEBHOOK) $(ENABLE_EXPERIMENTAL)
./scripts/install-gateway.sh $(GW_API_VERSION) $(ENABLE_EXPERIMENTAL)
kubectl apply -f $(CRDS)
kubectl apply -f $(NGF_MANIFEST)

.PHONY: prepare-ngf-dependencies-with-plus
prepare-ngf-dependencies-with-plus: update-ngf-manifest-with-plus ## Install NGF dependencies with Plus on configured kind cluster
./scripts/install-gateway.sh $(GW_API_VERSION) $(INSTALL_WEBHOOK)
./scripts/install-gateway.sh $(GW_API_VERSION)
kubectl apply -f $(CRDS)
kubectl apply -f $(NGF_MANIFEST)

Expand Down Expand Up @@ -120,7 +119,7 @@ uninstall-ngf: uninstall-k8s-components undo-manifests-update ## Uninstall NGF o
.PHONY: uninstall-k8s-components
uninstall-k8s-components: ## Uninstall installed components on configured kind cluster
-kubectl delete -f $(NGF_MANIFEST)
./scripts/uninstall-gateway.sh $(GW_API_VERSION) $(INSTALL_WEBHOOK) $(ENABLE_EXPERIMENTAL)
./scripts/uninstall-gateway.sh $(GW_API_VERSION) $(ENABLE_EXPERIMENTAL)
kubectl delete clusterrole nginx-gateway-provisioner
kubectl delete clusterrolebinding nginx-gateway-provisioner

Expand Down
1 change: 0 additions & 1 deletion conformance/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,6 @@ update-ngf-manifest Update the NGF deployment manifest image na
| SERVICE_MANIFEST | ../deploy/manifests/service/nodeport.yaml | The location of the NGF Service manifest |
| STATIC_MANIFEST | provisioner/static-deployment.yaml | The location of the NGF static deployment manifest |
| PROVISIONER_MANIFEST | provisioner/provisioner.yaml | The location of the NGF provisioner manifest |
| INSTALL_WEBHOOK | false | Install the Gateway API Validating Webhook. Necessary for Kubernetes versions < 1.25. |
| ENABLE_EXPERIMENTAL | false | Enable experimental features. Installs the Gateway APIs from the experimental channel and enables any supported experimental features in NGF. |

### Step 1 - Create a kind Cluster
Expand Down
17 changes: 2 additions & 15 deletions conformance/scripts/install-gateway.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,33 +5,20 @@ if [ -z $1 ]; then
exit 1
fi

if [ -z $2 ]; then
echo "install webhook argument not set; exiting"
exit 1
fi

if [ $1 == "main" ]; then
temp_dir=$(mktemp -d)
cd ${temp_dir}
curl -s https://codeload.github.com/kubernetes-sigs/gateway-api/tar.gz/main | tar -xz --strip=2 gateway-api-main/config
if [ $3 == "true" ]; then
if [ $2 == "true" ]; then
kubectl apply -f crd/experimental
else
kubectl apply -f crd/standard
fi
if [ $2 == "true" ]; then
kubectl apply -f webhook
kubectl wait --for=condition=available --timeout=60s deployment gateway-api-admission-server -n gateway-system
fi
rm -rf ${temp_dir}
else
if [ $3 == "true" ]; then
if [ $2 == "true" ]; then
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v$1/experimental-install.yaml
else
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v$1/standard-install.yaml
fi
if [ $2 == "true" ]; then
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v$1/webhook-install.yaml
kubectl wait --for=condition=available --timeout=60s deployment gateway-api-admission-server -n gateway-system
fi
fi
14 changes: 2 additions & 12 deletions conformance/scripts/uninstall-gateway.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,31 +5,21 @@ if [ -z $1 ]; then
exit 1
fi

if [ -z $2 ]; then
echo "install webhook argument not set; exiting"
exit 1
fi

if [ $1 == "main" ]; then
temp_dir=$(mktemp -d)
cd ${temp_dir}
curl -s https://codeload.github.com/kubernetes-sigs/gateway-api/tar.gz/main | tar -xz --strip=2 gateway-api-main/config
if [ $3 == "true" ]; then
if [ $2 == "true" ]; then
kubectl delete -f crd/experimental
else
kubectl delete -f crd/standard
fi
if [ $2 == "true" ]; then
kubectl delete -f webhook
fi
rm -rf ${temp_dir}
else
if [ $3 == "true" ]; then
if [ $2 == "true" ]; then
kubectl delete -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v$1/experimental-install.yaml
else
kubectl delete -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v$1/standard-install.yaml
fi
if [ $2 == "true" ]; then
kubectl delete -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v$1/webhook-install.yaml
fi
fi
31 changes: 0 additions & 31 deletions design/resource-validation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,27 +37,6 @@ To help the implementations with the validation, the Gateway API already include
OpenAPI schema validation. For example, the field X must be specified when type is set to Y; or X must be nil if
Y is not Z. Note: Kubernetes API server enforces this validation. To bypass it, a user needs to change the CRDs.

#### For Kubernetes 1.23 and 1.24

- *The webhook validation*. This validation is written in go and ran as part of the webhook, which is included in the
Gateway API installation files. The validation covers additional logic, not possible to implement in the OpenAPI
schema validation.
It does not repeat the OpenAPI schema validation from the CRDs. Note: a user can bypass this validation if the webhook
is not installed.

However, the built-in validation rules do not cover all validation needs of NGF:

- The rules are not enough for NGINX. For example, the validation rule for the
`value` of the path in a path-based routing rule allows symbols like `;`, `{`
and `}`, which can break NGINX configuration for the
corresponding [location](https://nginx.org/en/docs/http/ngx_http_core_module.html#location) block.
- The rules don't cover unsupported field cases. For example, the webhook does not know which filters are implemented by
NGF, thus it cannot generate an appropriate error for NGF.

Additionally, as mentioned in [GEP-922](https://gateway-api.sigs.k8s.io/geps/gep-922/#implementers),
"implementers must not rely on webhook or CRD validation as a security mechanism. If field values need to be escaped to
secure an implementation, both webhook and CRD validation can be bypassed and cannot be relied on."

## Requirements

Design a validation mechanism for Gateway API resources.
Expand Down Expand Up @@ -120,16 +99,6 @@ following methods in order of their appearance in the table.
| CRD validation | OpenAPI and CEL validation | Kubernetes API server | Structure, field values | Kubernetes API server returns any errors a response for an API call. | Yes, if the CRDs are modified. |
| NGF-specific validation | Go code | NGF control plane | Field values | Errors are reported in the status of a resource after its creation/modification. | No |


#### For Kubernetes 1.23 and 1.24

| Name | Type | Component | Scope | Feedback loop for errors | Can be bypassed? |
|------------------------------|---------|-----------------------|-------------------------|----------------------------------------------------------------------------------|--------------------------------------------------------------------------------------|
| CRD validation | OpenAPI | Kubernetes API server | Structure, field values | Kubernetes API server returns any errors a response for an API call. | Yes, if the CRDs are modified. |
| Webhook validation | Go code | Gateway API webhook | Field values | Kubernetes API server returns any errors a response for an API call. | Yes, if the webhook is not installed, misconfigured, or running a different version. |
| NGF-specific validation | Go code | NGF control plane | Field values | Errors are reported in the status of a resource after its creation/modification. | No |


Notes:

- The amount and the extent of the validation should allow multiple application developers to share a single NGF (User
Expand Down
8 changes: 0 additions & 8 deletions site/content/includes/installation/install-gateway-api-resources.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,3 @@ additional features provided by the experimental channel. To install from the ex
```shell
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/experimental-install.yaml
```

If you are running on Kubernetes 1.23 or 1.24, you also need to install the validating webhook. To do so, run:

```shell
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/webhook-install.yaml
```

{{< important >}}The validating webhook is not needed if you are running Kubernetes 1.25+. Validation is done using CEL on the CRDs. See the [resource validation doc]({{< relref "/overview/resource-validation.md" >}}) for more information.{{< /important >}}
6 changes: 0 additions & 6 deletions site/content/includes/installation/uninstall-gateway-api-resources.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,3 @@ docs: "DOCS-1436"
```shell
kubectl delete -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/experimental-install.yaml
```

If you are running on Kubernetes 1.23 or 1.24, you also need to delete the validating webhook. To do so, run:

```shell
kubectl delete -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/webhook-install.yaml
```
13 changes: 0 additions & 13 deletions site/content/installation/installing-ngf/manifests.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -147,19 +147,6 @@ To upgrade NGINX Gateway Fabric and get the latest features and improvements, ta
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/experimental-install.yaml
```

- If you are running on Kubernetes 1.23 or 1.24, you also need to update the validating webhook:

```shell
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/webhook-install.yaml
```

- If you are running on Kubernetes 1.25 or newer and have the validating webhook installed, you should remove the
webhook:

```shell
kubectl delete -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/webhook-install.yaml
```

1. **Upgrade NGINX Gateway Fabric CRDs:**
- To upgrade the Custom Resource Definitions (CRDs), run:

Expand Down
34 changes: 9 additions & 25 deletions site/content/overview/resource-validation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,36 +33,20 @@ The HTTPRoute "coffee" is invalid: spec.hostnames[0]: Invalid value: "cafe.!@#$%

{{< note >}}While unlikely, bypassing this validation step is possible if the Gateway API CRDs are modified to remove the validation. If this happens, Step 4 will reject any invalid values (from NGINX perspective).{{< /note >}}

### Step 2 - CEL or Webhook validation by Kubernetes
### Step 2 - CEL validation by Kubernetes API Server

- **Kubernetes 1.25 and later - CEL validation by Kubernetes API Server**
The Kubernetes API server validates Gateway API resources using CEL validation embedded in the Gateway API CRDs. It validates Gateway API resources using advanced rules unavailable in the OpenAPI schema validation. For example, if you create a Gateway resource with a TCP listener that configures a hostname, the CEL validation will reject it with the following error:

The Kubernetes API server validates Gateway API resources using CEL validation embedded in the Gateway API CRDs. It validates Gateway API resources using advanced rules unavailable in the OpenAPI schema validation. For example, if you create a Gateway resource with a TCP listener that configures a hostname, the CEL validation will reject it with the following error:

```shell
kubectl apply -f some-gateway.yaml
```

```text
The Gateway "some-gateway" is invalid: spec.listeners: Invalid value: "array": hostname must not be specified for protocols ['TCP', 'UDP']
```

More information on CEL in Kubernetes can be found [here](https://kubernetes.io/docs/reference/using-api/cel/).


- **Kubernetes 1.23 and 1.24 - Webhook validation by Gateway API Webhook**

The validating webhook must be [installed for these Kubernetes versions]({{< relref "installation/installing-ngf/helm.md#installing-the-gateway-api-resources" >}}). It validates Gateway API resources using advanced rules unavailable in the OpenAPI schema validation. For example, if you create a Gateway resource with a TCP listener that configures a hostname, the webhook will reject it with the following error:
```shell
kubectl apply -f some-gateway.yaml
```

```shell
kubectl apply -f some-gateway.yaml
```
```text
The Gateway "some-gateway" is invalid: spec.listeners: Invalid value: "array": hostname must not be specified for protocols ['TCP', 'UDP']
```

```text
Error from server: error when creating "some-gateway.yaml": admission webhook "validate.gateway.networking.k8s.io" denied the request: spec.listeners[1].hostname: Forbidden: should be empty for protocol TCP
```
More information on CEL in Kubernetes can be found [here](https://kubernetes.io/docs/reference/using-api/cel/).

{{< note >}}Bypassing this validation step is possible if the webhook is not running in the cluster. If this happens, Step 3 will reject the invalid values.{{< /note >}}

### Step 3 - Validation by NGINX Gateway Fabric

Expand Down
Loading
Loading