add eni-multi-ip documents

AliyunContainerService · Mar 19, 2019 · 73af78a · 73af78a
1 parent 7da5160
commit 73af78a
Show file tree

Hide file tree

Showing 4 changed files with 88 additions and 15 deletions.
diff --git a/README-zh_CN.md b/README-zh_CN.md
@@ -8,20 +8,33 @@ CNI plugin for alibaba cloud VPC/ENI
 ## 安装Kubernetes
 使用kubeadm的指导文档 https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/ 来创建集群
 
-安装好了之后要将iptables的policy换成ACCEPT，`iptables -P FORWARD ACCEPT`
+安装好了之后要:
+* 将iptables的policy换成ACCEPT，`iptables -P FORWARD ACCEPT`。
+* 检查节点上的"rp_filter"内核参数，并在每个节点上将其设置为"0"。
 
 通过`kubectl get cs`验证集群安装完成
 
 ## 安装terway插件
 
-修改[terway.yml](./terway.yml)文件中的eni.conf的配置中的授权和网段配置，以及Network的网段配置，然后通过`kubectl apply -f terway.conf`来安装terway插件。
+Terway有两种安装模式：
+
+* VPC模式
+
+	VPC模式，使用Aliyun VPC路由来打通网络，可以使用独立ENI给Pod，安装方式：<br />
+	修改[terway.yml](./terway.yml)文件中的eni.conf的配置中的授权和网段配置，以及Network的网段配置，然后通过`kubectl apply -f terway.conf`来安装terway插件。
+
+* ENI多IP模式
+
+	ENI多IP模式，使用Aliyun ENI的辅助IP来打通网络，不受VPC的路由条目限制，安装方式：<br />
+	修改[terway-multiip.yml](./terway-multiip.yml)文件中的eni.conf的配置中的授权和网段配置，以及Network的网段配置，然后通过`kubectl apply -f terway.conf`来安装terway插件。
+
 
 使用`kubectl get ds terway`看到插件在每个节点上都运行起来后，表明插件安装成功。
 
 ## 验证terway的功能
 
 ### 一般VPC网络的容器
-在容器没有做任何特殊配置时，terway会通过在节点上的podCidr中去分配地址然后配置给容器。
+在VPC安装模式下，在容器没有做任何特殊配置时，terway会通过在节点上的podCidr中去分配地址然后配置给容器。
 例如：
 
 ```
@@ -47,9 +60,9 @@ If you don't see a command prompt, try pressing enter.
        valid_lft forever preferred_lft forever
 ```   
 
-### 使用ENI弹性网卡获得等同于底层网络的性能
+#### 使用ENI弹性网卡获得等同于底层网络的性能
 
-在Pod的其中一个container的`requests`中增加对eni的需求： `aliyun/eni: 1`， 下面的例子将创建一个Nginx Pod，并分配一个ENI
+在VPC安装模式下，在Pod的其中一个container的`requests`中增加对eni的需求： `aliyun/eni: 1`， 下面的例子将创建一个Nginx Pod，并分配一个ENI
 
 ```
 apiVersion: v1
@@ -88,6 +101,29 @@ spec:
        valid_lft forever preferred_lft forever
 ```
 
+#### ENI辅助IP的容器：
+
+在ENI多IP安装模式下，Terway会通过创建和分配ENI和ENI网卡上的辅助IP地址给Pod使用，Pod上的IP地址将和VPC和VSwitch的IP地址相同段，例如：
+
+```
+[root@iZj6c86lmr8k9rk78ju0ncZ ~]# kubectl get pod -o wide
+NAME                     READY   STATUS    RESTARTS   AGE   IP              NODE                                 NOMINATED NODE
+nginx-64f497f8fd-ckpdm   1/1     Running   0          4d    192.168.0.191   cn-hangzhou.i-j6c86lmr8k9rk78ju0nc   <none>
+[root@iZj6c86lmr8k9rk78ju0ncZ ~]# kubectl get node -o wide cn-hangzhou.i-j6c86lmr8k9rk78ju0nc
+NAME                                 STATUS   ROLES    AGE   VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION              CONTAINER-RUNTIME
+cn-hangzhou.i-j6c86lmr8k9rk78ju0nc   Ready    <none>   12d   v1.11.5   192.168.0.154   <none>        CentOS Linux 7 (Core)   3.10.0-693.2.2.el7.x86_64   docker://17.6.2
+[root@iZj6c86lmr8k9rk78ju0ncZ ~]# kubectl exec -it nginx-64f497f8fd-ckpdm bash
+root@nginx-64f497f8fd-ckpdm:/# ip addr show
+1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
+    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
+    inet 127.0.0.1/8 scope host lo
+       valid_lft forever preferred_lft forever
+3: eth0@if106: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
+    link/ether 4a:60:eb:97:f4:07 brd ff:ff:ff:ff:ff:ff link-netnsid 0
+    inet 192.168.0.191/32 brd 192.168.0.191 scope global eth0
+       valid_lft forever preferred_lft forever
+```
+
 ### 使用NetworkPolicy来限制容器间访问
 
 Terway插件兼容标准的K8S中的NetworkPolicy来控制容器间的访问，例如：

diff --git a/README.md b/README.md
@@ -9,21 +9,35 @@ English | [简体中文](./README-zh_CN.md)
 ### Install Kubernetes
 Install Kubernetes via Kubeadm: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/
 
-After setup kubernetes cluster. Change `iptables` `Forward` default policy to `ACCEPT` on every node of cluster: `iptables -P FORWARD ACCEPT`.
+After setup kubernetes cluster.
+* Change `iptables` `Forward` default policy to `ACCEPT` on every node of cluster: `iptables -P FORWARD ACCEPT`.
+* Check the `rp_filter` in sysctl parameters, set them to "0" on every node of cluster.
 
 Make sure cluster up and healthy by `kubectl get cs`.
 
 ### Install Terway network plugin
 
-Replace `Network` and `AccessKey/AccessKeySecret` in [terway.yml](./terway.yml) with your cluster pod subnet and aliyun openapi credentials. Then use `kubectl apply -f terway.yml` to install Terway into kubernetes cluster.
+<br />
+Terway plugin have two installation modes
+
+* VPC Mode
+
+	VPC Mode, Using `Aliyun VPC` route table to connect the pods. Can assign dedicated ENI to Pod. Install method: <br />
+	Replace `Network` and `AccessKey/AccessKeySecret` in [terway.yml](./terway.yml) with your cluster pod subnet and aliyun openapi credentials. Then use `kubectl apply -f terway.yml` to install Terway into kubernetes cluster.
+
+* ENI Secondary IP Mode
+
+	ENI Secondary IP Mode, Using `Aliyun ENI's secondary ip` to connect the pods. This mode not limited by VPC route tables quotation. Install method: <br />
+	Replace `Network` and `AccessKey/AccessKeySecret` in [terway-multiip.yml](./terway-multiip.yml) with your cluster pod subnet and aliyun openapi credentials. Then use `kubectl apply -f terway.yml` to install Terway into kubernetes cluster.
+
 
 Using `kubectl get ds terway -n kube-system` to watch plugin launching. Plugin install completed while terway daemonset available pods equal to nodes.
 
 ### Terway network plugin usage:
 
 #### Vpc network container:
 
-Terway will config pod's address using node's `podCidr` when pod not have any especial config. eg:
+On VPC installation mode, Terway will config pod's address using node's `podCidr` when pod not have any especial config. eg:
 
 ```
 [root@iZj6c86lmr8k9rk78ju0ncZ ~]# kubectl run -it --rm --image busybox busybox
@@ -49,7 +63,7 @@ If you don't see a command prompt, try pressing enter.
 ```   
 
 #### Using ENI network interface to get the performance equivalent to the underlying network.
-Config `eni` request `aliyun/eni: 1` in one container of pod. The following example will create an Nginx Pod and assign an ENI:
+On VPC installation mode, Config `eni` request `aliyun/eni: 1` in one container of pod. The following example will create an Nginx Pod and assign an ENI:
 
 ```
 apiVersion: v1
@@ -86,6 +100,29 @@ spec:
        valid_lft forever preferred_lft forever
 ```
 
+#### ENI Secondary IP Pod：
+
+On ENI secondary IP installation mode, Terway will create & allocate ENI secondary IP for pod. The IP of pod will in same IP Range:
+
+```
+[root@iZj6c86lmr8k9rk78ju0ncZ ~]# kubectl get pod -o wide
+NAME                     READY   STATUS    RESTARTS   AGE   IP              NODE                                 NOMINATED NODE
+nginx-64f497f8fd-ckpdm   1/1     Running   0          4d    192.168.0.191   cn-hangzhou.i-j6c86lmr8k9rk78ju0nc   <none>
+[root@iZj6c86lmr8k9rk78ju0ncZ ~]# kubectl get node -o wide cn-hangzhou.i-j6c86lmr8k9rk78ju0nc
+NAME                                 STATUS   ROLES    AGE   VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION              CONTAINER-RUNTIME
+cn-hangzhou.i-j6c86lmr8k9rk78ju0nc   Ready    <none>   12d   v1.11.5   192.168.0.154   <none>        CentOS Linux 7 (Core)   3.10.0-693.2.2.el7.x86_64   docker://17.6.2
+[root@iZj6c86lmr8k9rk78ju0ncZ ~]# kubectl exec -it nginx-64f497f8fd-ckpdm bash
+root@nginx-64f497f8fd-ckpdm:/# ip addr show
+1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
+    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
+    inet 127.0.0.1/8 scope host lo
+       valid_lft forever preferred_lft forever
+3: eth0@if106: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
+    link/ether 4a:60:eb:97:f4:07 brd ff:ff:ff:ff:ff:ff link-netnsid 0
+    inet 192.168.0.191/32 brd 192.168.0.191 scope global eth0
+       valid_lft forever preferred_lft forever
+```
+
 #### Using network policy to limit accessible between containers.
 
 The Terway plugin is compatible with NetworkPolicy in the standard K8S to control access between containers, for example:

diff --git a/terway-multiip.yml b/terway-multiip.yml
@@ -106,7 +106,7 @@ spec:
       hostNetwork: true
       initContainers:
       - name: terway-init
-        image: registry.aliyuncs.com/wangbs/terway
+        image: registry.aliyuncs.com/acs/terway:v1.0.10.0-g7da5160-aliyun
         imagePullPolicy: Always
         securityContext:
           privileged: true
@@ -131,7 +131,7 @@ spec:
           name: host-root
       containers:
       - name: terway
-        image: registry.aliyuncs.com/wangbs/terway
+        image: registry.aliyuncs.com/acs/terway:v1.0.10.0-g7da5160-aliyun
         imagePullPolicy: Always
         command: ['/usr/bin/terwayd', '-log-level', 'debug', '-daemon-mode', 'ENIMultiIP']
         securityContext:
@@ -157,7 +157,7 @@ spec:
         - mountPath: /var/lib/kubelet/device-plugins
           name: device-plugin-path
       - name: policy
-        image: registry.aliyuncs.com/wangbs/terway
+        image: registry.aliyuncs.com/acs/terway:v1.0.10.0-g7da5160-aliyun
         imagePullPolicy: Always
         command: ["/bin/policyinit.sh"]
         env:

diff --git a/terway.yml b/terway.yml
@@ -104,7 +104,7 @@ spec:
       hostNetwork: true
       initContainers:
       - name: terway-init
-        image: registry.aliyuncs.com/acs/terway:v0.1.0
+        image: registry.aliyuncs.com/acs/terway:v1.0.10.0-g7da5160-aliyun
         imagePullPolicy: Always
         securityContext:
           privileged: true
@@ -120,7 +120,7 @@ spec:
           name: lib-modules
       containers:
       - name: terway
-        image: registry.aliyuncs.com/acs/terway:v0.1.0
+        image: registry.aliyuncs.com/acs/terway:v1.0.10.0-g7da5160-aliyun
         imagePullPolicy: Always
         securityContext:
           privileged: true
@@ -143,7 +143,7 @@ spec:
         - mountPath: /var/lib/kubelet/device-plugins
           name: device-plugin-path
       - name: policy
-        image: registry.aliyuncs.com/acs/terway:v0.1.0
+        image: registry.aliyuncs.com/acs/terway:v1.0.10.0-g7da5160-aliyun
         command: ["/bin/policyinit.sh"]
         imagePullPolicy: Always
         env: