kubernetes-sigs · k8s-ci-robot · Jun 13, 2019 · Jun 13, 2019
diff --git a/docs/azure-loadbalancer.md b/docs/azure-loadbalancer.md
@@ -1,6 +1,6 @@
 # Azure LoadBalancer
 
-The way Azure defines a LoadBalancer is different from GCE or AWS. Azure's LB can have multiple frontend IP refs. GCE and AWS only allow one, if you want more, you would need multiple LB's. Since Public IP's are not part of the LB in Azure an NSG is not part of LB in Azure either. However, you cannot delete them in parallel, a Public IP can only be deleted after LB's frontend IP ref is removed. 
+The way Azure defines a LoadBalancer is different from GCE or AWS. Azure's LB can have multiple frontend IP refs. GCE and AWS only allow one, if you want more, you would need multiple LB's. Since Public IP's are not part of the LB in Azure an NSG is not part of LB in Azure either. However, you cannot delete them in parallel, a Public IP can only be deleted after LB's frontend IP ref is removed.
 
 The different Azure Resources such as LB, Public IP, and NSG are the same tier of Azure resources and circular dependencies need to be avoided. In another words, they should only depend on service state.
 
@@ -9,17 +9,17 @@ By default the basic SKU is selected for a load balancer. Services can be annota
 
 Below is a list of annotations supported for Kubernetes services with type `LoadBalancer`:
 
-| Annotaion                                                    | Value                        | Description                                                  |
-| ------------------------------------------------------------ | ---------------------------- | ------------------------------------------------------------ |
-| `service.beta.kubernetes.io/azure-load-balancer-internal`    | `true `or `false`            | Specify whether the load balancer should be internal. It’s defaulting to public if not set. |
-| `service.beta.kubernetes.io/azure-load-balancer-internal-subnet` | Name of the subnet           | Specify which subnet the internal load balancer should be bound to. It’s defaulting to the subnet configured in cloud config file if not set. |
-| `service.beta.kubernetes.io/azure-load-balancer-mode`        | `auto`, `{name1},{name2}`    | Specify the Azure load balancer selection algorithm based on availability sets. There are currently three possible load balancer selection modes : default, auto or "{name1}, {name2}". This is only working for basic LB (see below for how it works) |
-| `service.beta.kubernetes.io/azure-dns-label-name`            | Name of the DNS label        | Specify the DNS label name for the service.                  |
-| `service.beta.kubernetes.io/azure-shared-securityrule`       | `true` or `false`            | Specify that the service should be exposed using an Azure security rule that may be shared with other service, trading specificity of rules for an increase in the number of services that can be exposed. This relies on the Azure "augmented security rules" feature. |
-| `service.beta.kubernetes.io/azure-load-balancer-resource-group` | Name of the resource group   | Specify the resource group of load balancer objects that are not in the same resource group as the cluster. |
-| `service.beta.kubernetes.io/azure-allowed-service-tags`      | List of allowed service tags | Specify a list of allowed [service tags](https://docs.microsoft.com/en-us/azure/virtual-network/security-overview#service-tags) separated by comma. |
-| `service.beta.kubernetes.io/azure-load-balancer-tcp-idle-timeout` | TCP idle timeouts in minutes | Specify the time, in minutes, for TCP connection idle timeouts to occur on the load balancer. Default and minimum value is 4. Maximum value is 30. Must be an integer. |
-| `service.beta.kubernetes.io/azure-load-balancer-mixed-protocols` | `true` or `false`            | Specify whether both TCP and UDP protocols should be created for the service. (This is not allowed from Kubernetes API) |
+| Annotaion                                                    | Value                        | Description                                                  | Kubernetes Version |
+| ------------------------------------------------------------ | ---------------------------- | ------------------------------------------------------------ |------|
+| `service.beta.kubernetes.io/azure-load-balancer-internal`    | `true `or `false`            | Specify whether the load balancer should be internal. It’s defaulting to public if not set. | v1.10.0 and later |
+| `service.beta.kubernetes.io/azure-load-balancer-internal-subnet` | Name of the subnet           | Specify which subnet the internal load balancer should be bound to. It’s defaulting to the subnet configured in cloud config file if not set. | v1.10.0 and later |
+| `service.beta.kubernetes.io/azure-load-balancer-mode`        | `auto`, `{name1},{name2}`    | Specify the Azure load balancer selection algorithm based on availability sets. There are currently three possible load balancer selection modes : default, auto or "{name1}, {name2}". This is only working for basic LB (see below for how it works) |  v1.10.0 and later |
+| `service.beta.kubernetes.io/azure-dns-label-name`            | Name of the DNS label        | Specify the DNS label name for the service.                  | v1.10.0 and later |
+| `service.beta.kubernetes.io/azure-shared-securityrule`       | `true` or `false`            | Specify that the service should be exposed using an Azure security rule that may be shared with other service, trading specificity of rules for an increase in the number of services that can be exposed. This relies on the Azure "augmented security rules" feature. | v1.10.0 and later |
+| `service.beta.kubernetes.io/azure-load-balancer-resource-group` | Name of the resource group   | Specify the resource group of load balancer objects that are not in the same resource group as the cluster. | v1.10.0 and later |
+| `service.beta.kubernetes.io/azure-allowed-service-tags`      | List of allowed service tags | Specify a list of allowed [service tags](https://docs.microsoft.com/en-us/azure/virtual-network/security-overview#service-tags) separated by comma. | v1.11.0 and later |
+| `service.beta.kubernetes.io/azure-load-balancer-tcp-idle-timeout` | TCP idle timeouts in minutes | Specify the time, in minutes, for TCP connection idle timeouts to occur on the load balancer. Default and minimum value is 4. Maximum value is 30. Must be an integer. |  v1.11.4, v1.12.0 and later |
+| `service.beta.kubernetes.io/azure-load-balancer-mixed-protocols` | `true` or `false`            | Specify whether both TCP and UDP protocols should be created for the service. (This is not allowed from Kubernetes API) | v1.12.7, v1.13.4 and later |
 
 ### Load balancer selection modes
 
@@ -39,7 +39,7 @@ Azure cloud provider supports both `basic` and `standard` SKU load balancers, wh
 
 > Note that the public IPs used in load balancer frontend configurations should be same SKU. That is standard public IP for standard load balancer, while basic public IP for basic load balancer.
 
-Azure doesn’t support a network interface joining load balancers with different SKUs, hence migration dynamically between them is not supported. 
+Azure doesn’t support a network interface joining load balancers with different SKUs, hence migration dynamically between them is not supported.
 
 > If you do require migration, please delete all services with type `LoadBalancer` (or change to other type)
 
@@ -60,11 +60,10 @@ Because the load balancer in a Kubernetes cluster is managed by Azure cloud prov
 >
 > * In the context of outbound connectivity, a single standalone VM, all the VM's in an Availability Set, all the instances in a VMSS behave as a group. This means, if a single VM in an Availability Set is associated with a Standard SKU, all VM instances within this Availability Set now behave by the same rules as if they are associated with Standard SKU, even if an individual instance is not directly associated with it.
 >
-> * Public IP's used as instance-level public IP are mutually exclusive with outbound rules.  
+> * Public IP's used as instance-level public IP are mutually exclusive with outbound rules.
 
 Here is the recommend way to define the [outbound rules](https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-outbound-rules-overview) when using seperate provisioning tools:
 
 * Create a separate IP (or multiple IPs for scale) in standard SKU for outbound rules. Make use of the [allocatedOutboundPorts](https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-outbound-rules-overview#snatports) parameter to allocate sufficient ports for your desired scenario scale.
 * Create a separate pool definition for outbound, and ensure all virtual machines or VMSS virtual machines are in this pool. Azure cloud provider will manage the load balancer rules with another pool, so that provisioning tools and Azure cloud provider won't affect each other.
 * Define inbound with load balancing rules and inbound NAT rules as needed, and set `disableOutboundSNAT` to true on the load balancing rule(s).  Don't rely on the side effect from these rules for outbound connectivity. It makes it messier than it needs to be and limits your options.  Use inbound NAT rules to create port forwarding mappings for SSH access to the VM's rather than burning public IPs per instance.
-
diff --git a/docs/cloud-provider-config.md b/docs/cloud-provider-config.md
@@ -19,6 +19,7 @@ Here is a config file sample:
     "vnetResourceGroup": "",
     "routeTableName": "<name>",
     "primaryAvailabilitySetName": "<name>",
+    "routeTableResourceGroup": "<name>",
     "cloudProviderBackoff": false,
     "useManagedIdentityExtension": false,
     "useInstanceMetadata": true
@@ -79,7 +80,10 @@ If more than one value is set, the order is `Managed Identity` > `Service Princi
 |useInstanceMetadata|Use instance metadata service where possible|Boolean value, default to false|
 |loadBalancerSku|Sku of Load Balancer and Public IP. Candidate values are: `basic` and `standard`.|Default to `basic`.|
 |excludeMasterFromStandardLB|ExcludeMasterFromStandardLB excludes master nodes from standard load balancer.|Boolean value, default to true.|
+|disableOutboundSNAT| Disable outbound SNAT for SLB | Default to false and avaible since v1.11.9, v1.12.7, v1.13.5 and v1.14.0|
 |maximumLoadBalancerRuleCount|Maximum allowed LoadBalancer Rule Count is the limit enforced by Azure Load balancer|Integer value, default to [148](https://github.com/kubernetes/kubernetes/blob/v1.10.0/pkg/cloudprovider/providers/azure/azure.go#L48)|
+|routeTableResourceGroup| The resource group name for routeTable | Default same as resourceGroup and available since v1.15.0 |
+|cloudConfigType| The cloud configure type for Azure cloud provider. Supported values are file, secret and merge.| Default to `merge`.  and available since v1.15.0 |
 
 ### primaryAvailabilitySetName
 
@@ -101,7 +105,66 @@ Master nodes would not add to the backends of Azure loadbalancer (ALB) if `exclu
 
 By default, if nodes are labeled with `node-role.kubernetes.io/master`, they would also be excluded from ALB. If you want adding the master nodes to ALB, `excludeMasterFromStandardLB` should be set to false and label `node-role.kubernetes.io/master` should be removed if it has already been applied.
 
-# Azure Stack Configuration
+### Setting Azure cloud provider from Kubernetes secrets
+
+Since v1.15.0, Azure cloud provider supports reading the cloud config from Kubernetes secrets. The secret is a serialized version of `azure.json` file with key `cloud-config`. The secret should be put in `kube-system` namespace and its name should be `azure-cloud-provider`.
+
+To enable this feature, set `cloudConfigType` to `secret` or `merge` (default is `mergg`). All supported values for this option are:
+
+- `file`: The cloud provider configuration is read from cloud-config file.
+- `secret`: the cloud provider configuration must be overridden by the secret.
+- `merge`: the cloud provider configuration can be optionally overridden by a secret when it is set explicitly in the secret, this is default value.
+
+Since Azure cloud provider would read Kubernetes secrets, the following RBAC should also be configured:
+
+```yaml
+---
+apiVersion: rbac.authorization.k8s.io/v1beta1
+kind: ClusterRole
+metadata:
+  labels:
+    kubernetes.io/cluster-service: "true"
+  name: system:azure-cloud-provider-secret-getter
+rules:
+- apiGroups: [""]
+  resources: ["secrets"]
+  verbs:
+  - get
+---
+apiVersion: rbac.authorization.k8s.io/v1beta1
+kind: ClusterRoleBinding
+metadata:
+  labels:
+    kubernetes.io/cluster-service: "true"
+  name: system:azure-cloud-provider-secret-getter
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: system:azure-cloud-provider-secret-getter
+subjects:
+- kind: ServiceAccount
+  name: azure-cloud-provider
+  namespace: kube-system
+```
+
+## Run Kubelet without Azure identity
+
+When running Kubelet with kube-controller-manager, it also supports running without Azure identity since v1.15.0.
+
+Both kube-controller-manager and kubelet should configure `--cloud-provider=azure --cloud-config=/etc/kubernetes/azure.json`, but the contents for `azure.json` are different:
+
+(1) For kube-controller-manager, refer the above part for setting `azure.json`.
+
+(2) For kubelet, `useInstanceMetadata` is required to be `true` and Azure identities are not required. A sample for Kubelet's azure.json is
+
+```json
+{
+  "useInstanceMetadata": true,
+  "vmType": "vmss"
+}
+```
+
+## Azure Stack Configuration
 
 Azure Stack has different API endpoints, depending on the Azure Stack deployment. These need to be provided to the Azure SDK and currently this is done by adding an extra `json` file with the arguments, as well as an environment variable pointing to this file.
 
@@ -120,7 +183,7 @@ When `cloud: AzureStackCloud`, the extra environment variable used by the Azure
 
 The configuration parameters of this file:
 
-```
+```json
 {
   "name": "AzureStackCloud",
   "managementPortalURL": "...",