-
Notifications
You must be signed in to change notification settings - Fork 67
Add docs for enabling Istio native nftables feature #1122
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 6 commits
5d59776
bf8c65e
92b36e1
ef831d2
e27c7e6
329ec6e
bacfac1
2c68a3a
364bab8
1057712
1b488f7
8a2d4a7
1340b2b
f89ca91
f7de57d
c48d831
686fb07
28e913c
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| apiVersion: sailoperator.io/v1 | ||
| kind: Istio | ||
| metadata: | ||
| name: default | ||
| spec: | ||
| version: master | ||
| namespace: istio-system | ||
| updateStrategy: | ||
| type: InPlace | ||
| inactiveRevisionDeletionGracePeriodSeconds: 30 | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Lets remove these fields as they are anyways default. |
||
| values: | ||
| global: | ||
| nativeNftables: true | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| apiVersion: sailoperator.io/v1 | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Same.... |
||
| kind: IstioCNI | ||
| metadata: | ||
| name: default | ||
| spec: | ||
| version: master | ||
| namespace: istio-cni | ||
| values: | ||
| global: | ||
| nativeNftables: true | ||
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,237 @@ | ||||||
| [Return to Project Root](../README.md) | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Hey, can you please convert these files to ASCII doc? We now have all the docs under the docs folder in ASCII format
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. yes, I pushed another commit about adding a ascii one. I'm checking. If the conversion looks correct , I will remove the markdown file in another commit after review. |
||||||
|
|
||||||
| # Table of Contents | ||||||
|
|
||||||
| - [Istio nftables backend](#istio-nftables-backend) | ||||||
| - [Prerequisites](#prerequisites) | ||||||
| - [Installation](#installation) | ||||||
| - [Install with Sail Operator on OpenShift](#install-with-sail-operator-on-openshift) | ||||||
| - [Install in Ambient Mode](#install-in-ambient-mode) | ||||||
| - [Validation](#validation) | ||||||
| - [Upgrade](#upgrade) | ||||||
| - [Upgrade with Sail Operator on OpenShift](#upgrade-with-sail-operator-on-openshift) | ||||||
| - [Upgrade in Ambient Mode](#upgrade-in-ambient-mode) | ||||||
|
|
||||||
|
|
||||||
| ## Istio nftables backend | ||||||
|
|
||||||
| This document outlines the configuration steps for the nftables backend in Istio. As the official successor to iptables, nftables offers a modern, high-performance alternative for transparently redirecting traffic to and from the Envoy sidecar proxy. Many major Linux distributions are actively moving towards adopting native nftables support. | ||||||
| ### Prerequisites | ||||||
|
|
||||||
| - **nftables version**: Requires `nft` binary version 1.0.1 or later. | ||||||
|
|
||||||
| ### Installation | ||||||
|
|
||||||
| The support for native nftables when using Istio sidecar mode was implemented in the upstream istio release-1.27 [release note](https://github.com/istio/istio/blob/master/releasenotes/notes/nftables-sidecar.yaml). It is disabled by default. To enable it, you can set a feature flag as `values.global.nativeNftables=true`. For example, | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
|
||||||
| Installation with Istioctl | ||||||
|
|
||||||
| ```sh | ||||||
| istioctl install --set values.global.nativeNftables=true -y | ||||||
| ``` | ||||||
|
|
||||||
| Installation with Helm | ||||||
|
|
||||||
| ```sh | ||||||
| helm install istiod-canary istio/istiod \ | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Since we are talking about nftables, we can avoid using "istiod-canary"
Suggested change
|
||||||
| --set values.global.nativeNftables=true \ | ||||||
| -n istio-system | ||||||
| ``` | ||||||
|
|
||||||
| #### Install with Sail Operator on OpenShift | ||||||
|
|
||||||
| When you install an Istio resource with Sail Operator, you can create an instance of the `Istio` resource with `spec.values.global.nativeNftables=true`. | ||||||
|
|
||||||
| ```sh | ||||||
| apiVersion: sailoperator.io/v1 | ||||||
| kind: Istio | ||||||
| metadata: | ||||||
| name: default | ||||||
| spec: | ||||||
| version: v1.28-alpha.24646157 | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We can use 1.27 (or whatever is the latest supported version) here. |
||||||
| namespace: istio-system | ||||||
| updateStrategy: | ||||||
| type: InPlace | ||||||
| inactiveRevisionDeletionGracePeriodSeconds: 30 | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Since the updateStrategy section is anyways using the defaults, we can avoid specifying it in the yamls. |
||||||
| values: | ||||||
| global: | ||||||
| nativeNftables: true | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Actually you can remove this whole section since we are capturing the same info below. |
||||||
| ``` | ||||||
|
|
||||||
| This feature configures Istio to use the `nftables` backend instead of `iptables` for traffic redirection. | ||||||
|
|
||||||
| To enable the Istio native nftables feature, using the following steps: | ||||||
|
|
||||||
| 1. Create the `Subscription` object with a 1.27 channel. | ||||||
|
|
||||||
| ```sh | ||||||
| kubectl apply -f - <<EOF | ||||||
| apiVersion: operators.coreos.com/v1alpha1 | ||||||
| kind: Subscription | ||||||
| metadata: | ||||||
| name: sailoperator | ||||||
| namespace: openshift-operators | ||||||
| spec: | ||||||
| channel: "1.27-nightly" | ||||||
| installPlanApproval: Automatic | ||||||
| name: sailoperator | ||||||
| source: community-operators | ||||||
| sourceNamespace: openshift-marketplace | ||||||
| EOF | ||||||
| ``` | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. As this document is for Sail Operator, lets avoid any OpenShift specific notes. Please use appropriate mechanism for Kind cluster. |
||||||
|
|
||||||
| 2. Verify that the installation succeeded by inspecting the CSV status. | ||||||
|
|
||||||
| ```sh | ||||||
| NAME DISPLAY VERSION REPLACES PHASE | ||||||
| sailoperator.v1.27.0-nightly-2025-08-15 Sail Operator 1.27.0-nightly-2025-08-15 sailoperator.v1.27.0-nightly-2025-08-14 Succeeded | ||||||
| ``` | ||||||
|
|
||||||
| Succeeded should appear in the sailoperator CSV PHASE column. | ||||||
|
|
||||||
| 3. Create the `istio-system` and `istio-cni` namespaces. | ||||||
|
|
||||||
| ```sh | ||||||
| kubectl create namespace istio-system | ||||||
| kubectl create namespace istio-cni | ||||||
| ``` | ||||||
|
|
||||||
| 4. Create the `Istio` resource with `spec.values.global.nativeNftables=true`: | ||||||
|
|
||||||
| ```sh | ||||||
| apiVersion: sailoperator.io/v1 | ||||||
| kind: Istio | ||||||
| metadata: | ||||||
| name: default | ||||||
| spec: | ||||||
| version: v1.28-alpha.24646157 | ||||||
| namespace: istio-system | ||||||
| updateStrategy: | ||||||
| type: InPlace | ||||||
| inactiveRevisionDeletionGracePeriodSeconds: 30 | ||||||
| values: | ||||||
| global: | ||||||
| nativeNftables: true | ||||||
| ``` | ||||||
|
|
||||||
| 5. Create the `IstioCNI` resource with `spec.values.global.nativeNftables=true`: | ||||||
|
|
||||||
| ```sh | ||||||
| kind: IstioCNI | ||||||
| apiVersion: sailoperator.io/v1 | ||||||
| metadata: | ||||||
| name: default | ||||||
| spec: | ||||||
| namespace: istio-cni | ||||||
| version: v1.28-alpha.24646157 | ||||||
| values: | ||||||
| global: | ||||||
| nativeNftables: true | ||||||
| ``` | ||||||
|
|
||||||
| #### Install in Ambient Mode | ||||||
|
|
||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is this a TODO?
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. yes, the Ambient mode part install and upgrade is in progress. When that part is clear, I will update the upgrade task JIRA and this PR draft. |
||||||
| ### Validation | ||||||
|
|
||||||
| When using the `nftables` backend, you can verify the traffic redirection rules using the `nft list ruleset` command in the `istio-proxy` container. When using the nftables backend, You get the table inet rules in the istio-proxy container. The following example installs a sample application `curl` in a data plane namespace `test-ns`. | ||||||
|
|
||||||
| ```sh | ||||||
| kubectl create ns test-ns | ||||||
| kubectl label namespace test-ns istio-injection=enabled | ||||||
| kubectl apply -n test-ns -f https://raw.githubusercontent.com/istio/istio/refs/heads/master/samples/curl/curl.yaml | ||||||
| ``` | ||||||
|
|
||||||
| Attach a debug container and you will get the following rules in the `istio-proxy` container: | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
|
||||||
| ```sh | ||||||
| kubectl -n test-ns debug --image istio/base --profile netadmin --attach -t -i \ | ||||||
| "$(kubectl -n test-ns get pod -l app=curl -o jsonpath='{.items..metadata.name}')" | ||||||
| root@curl-6c88b89ddf-kbzn6:$ nft list ruleset | ||||||
|
|
||||||
| ``` | ||||||
|
|
||||||
| Verify the connectivity between two pods is working. For example, deploy a httpbin application using the following step: | ||||||
|
|
||||||
| ```sh | ||||||
| kubectl apply -n test-ns -f samples/httpbin/httpbin.yaml | ||||||
| kubectl exec -n test-ns "$(kubectl get pod -l app=curl -n test-ns -o jsonpath={.items..metadata.name})" -c curl -n test-ns -- curl http://httpbin.test-ns:8000/ip -s -o /dev/null -w "%{http_code}\n" | ||||||
|
|
||||||
| 200 | ||||||
| ``` | ||||||
|
|
||||||
| More guidelines: [Debugging Guidelines](https://github.com/istio/istio/tree/master/tools/istio-nftables/pkg#debugging-guidelines) | ||||||
|
|
||||||
| ### Upgrade | ||||||
|
|
||||||
| The migration of using the existing Istio iptables backend to nftables backend can be done by upgrading Istio. The following example installs an Istio control plane with the iptables backend and a sample application curl in a data plane namespace test-ns. | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
|
||||||
| ```sh | ||||||
| istioctl install -y | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. As this is a Sail Operator doc, lets use the installation mechanisms supported in Sail Operator. |
||||||
|
|
||||||
| kubectl create ns test-ns | ||||||
| kubectl label namespace test-ns istio-injection=enabled | ||||||
| kubectl apply -n test-ns -f samples/curl/curl.yaml | ||||||
| ``` | ||||||
|
|
||||||
| You may create another Istio control plane with a revision value and gradually migrate data plane traffic to the new revision Istio control plane. This canary upgrade approach is much safer than doing an in-place upgrade. For example, | ||||||
|
|
||||||
| 1. Install a canary version of Istio with the nftables backend. | ||||||
|
|
||||||
| ```sh | ||||||
| istioctl install --set revision=canary --set values.global.nativeNftables=true -y | ||||||
| ``` | ||||||
|
|
||||||
| 2. Upgrade the data plane and restart the deployment | ||||||
|
|
||||||
| ```sh | ||||||
| kubectl label namespace test-ns istio-injection- istio.io/rev=canary | ||||||
| kubectl rollout restart deployment -n test-ns | ||||||
| ``` | ||||||
|
|
||||||
| 3. Check the nftables backend running in the curl application pod. When using the nftables backend, You get the table inet rules in the istio-proxy container. For example, attach a debug container and run nft list ruleset command: | ||||||
|
|
||||||
| ```sh | ||||||
| kubectl -n test-ns debug --image istio/base --profile netadmin --attach -t -i \ | ||||||
| "$(kubectl -n test-ns get pod -l app=curl -o jsonpath='{.items..metadata.name}')" | ||||||
|
|
||||||
| root@curl-6c88b89ddf-kbzn6:$ nft list ruleset | ||||||
| ``` | ||||||
|
|
||||||
| 4. Verify the connectivity between two pods is working. For example, deploy a httpbin application using the following step: | ||||||
|
|
||||||
| ```sh | ||||||
| kubectl apply -n test-ns -f samples/httpbin/httpbin.yaml | ||||||
| kubectl exec -n test-ns "$(kubectl get pod -l app=curl -n test-ns -o jsonpath={.items..metadata.name})" -c curl -n test-ns -- curl http://httpbin.test-ns:8000/ip -s -o /dev/null -w "%{http_code}\n" | ||||||
|
|
||||||
| 200 | ||||||
| ``` | ||||||
|
|
||||||
| 5. After upgrading both the control plane and data plane, you can uninstall the old control plane in this example. | ||||||
|
|
||||||
| ```sh | ||||||
| istioctl uninstall --revision default -y | ||||||
| ``` | ||||||
|
|
||||||
| When upgrading Istio with the CNI node agent, you can install a canary version of Istio control plane and upgrade the istio-cni node agent separately. For example, there is an Istio CNI component running in the istio-cni namespace, you can upgrade and enable the nftables backend using the following steps: | ||||||
|
|
||||||
| 1. Install a canary version of Istiod control plane with the nftables backend. | ||||||
|
|
||||||
| ```sh | ||||||
| helm install istiod-canary istio/istiod \ | ||||||
| --set revision=canary \ | ||||||
| --set values.global.nativeNftables=true \ | ||||||
| -n istio-system | ||||||
| ``` | ||||||
|
|
||||||
| 2. Upgrade the CNI component separately from the revisioned control plane. | ||||||
|
|
||||||
| ```sh | ||||||
| helm upgrade istio-cni istio/cni \ | ||||||
| --set values.global.nativeNftables=true \ | ||||||
| -n istio-cni --wait | ||||||
| ``` | ||||||
|
|
||||||
| #### Upgrade with Sail Operator on OpenShift | ||||||
|
|
||||||
| #### Upgrade in Ambient mode | ||||||
|
|
||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
While this looks good. In the longer run, we should use templates and update it based on the args.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I dont see this file used anywhere in the PR.