```
-$a = Get-NetAdapter | where Name -Match HNSTransparent
-Rename-NetAdapter $a[0].Name -NewName HNSTransparent
-Stop-Service ovs-vswitchd -force; Disable-VMSwitchExtension "Cloudbase Open vSwitch Extension";
-ovs-vsctl --no-wait del-br br-ex
-ovs-vsctl --no-wait --may-exist add-br br-ex
-ovs-vsctl --no-wait add-port br-ex HNSTransparent -- set interface HNSTransparent type=internal
-ovs-vsctl --no-wait add-port br-ex $INTERFACE_ALIAS
-Enable-VMSwitchExtension "Cloudbase Open vSwitch Extension"; sleep 2; Restart-Service ovs-vswitchd
+
+
+
+ |
+Parameter
+ |
+ Default Value
+ |
+ Notes
+ |
+
+
+ |
+---
+ |
+ ---
+ |
+ ---
+ |
+
+
+ | -ManagementIP
+ |
+ N/A (required)
+ |
+ The IP address assigned to the Windows node. You can use ipconfig to find this.
+ |
+
+
+ | -NetworkMode
+ |
+ l2bridge
+ |
+ We're using overlay here
+ |
+
+
+ | -ClusterCIDR
+ |
+ 10.244.0.0/16
+ |
+ Refer to your cluster IP plan
+ |
+
+
+ | -ServiceCIDR
+ |
+ 10.96.0.0/12
+ |
+ Refer to your cluster IP plan
+ |
+
+
+ | -KubeDnsServiceIP
+ |
+ 10.96.0.10
+ |
+
+ |
+
+
+ | -InterfaceName
+ |
+ Ethernet
+ |
+ The name of the network interface of the Windows host. You can use ipconfig to find this.
+ |
+
+
+ | -LogDir
+ |
+ C:\k
+ |
+ The directory where kubelet and kube-proxy logs are redirected into their respective output files.
+ |
+
+
+
+Now you can view the Windows nodes in your cluster by running the following:
+
+```bash
+kubectl get nodes
```
-Besides of the above, setting up a Windows host is the same as the Linux host. Follow the steps from [here](https://github.com/openvswitch/ovn-kubernetes#k8s-minion-node-initializations).
-**Windows CNI Setup**
+{{< note >}}
+You may want to configure your Windows node components like kubelet and kube-proxy to run as services. View the services and background processes section under [troubleshooting](#troubleshooting) for additional instructions. Once you are running the node components as services, collecting logs becomes an important part of troubleshooting. View the [gathering logs](https://github.com/kubernetes/community/blob/master/sig-windows/CONTRIBUTING.md#gathering-logs) section of the contributing guide for further instructions.
+{{< /note >}}
-Today, Windows OVN&OVS CNI plugin is based on ovn_cni.exe which can be downloaded from [here](https://cloudbase.it/downloads/ovn_cni.exe). A sample of CNI config file is the following:
+### Public Cloud Providers
+
+#### Azure
+
+AKS-Engine can deploy a complete, customizable Kubernetes cluster with both Linux & Windows nodes. There is a step-by-step walkthrough available in the [docs on GitHub](https://github.com/Azure/aks-engine/blob/master/docs/topics/windows.md).
+
+#### GCP
+
+Users can easily deploy a complete Kubernetes cluster on GCE following this step-by-step walkthrough on [GitHub](https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/windows/README-GCE-Windows-kube-up.md)
+
+#### Deployment with kubeadm and cluster API
+
+Kubeadm is becoming the de facto standard for users to deploy a Kubernetes cluster. Windows node support in kubeadm will come in a future release. We are also making investments in cluster API to ensure Windows nodes are properly provisioned.
+
+#### Next Steps
+
+Now that you've configured a Windows worker in your cluster to run Windows containers you may want to add one or more Linux nodes as well to run Linux containers. Now you're ready to proceed to the next step to schedule Windows containers on your cluster.
+
+# User Guide: Scheduling Windows containers in Kubernetes
+
+## Objectives
+
+* Configure an example deployment to run Windows containers on the Windows node
+* (Optional) Configure an Active Directory Identity for your Pod using Group Managed Service Accounts (GMSA)
+
+## Before you begin
+
+* Create a Kubernetes cluster that includes a [master and a worker node running Windows Server](#UG-windows-nodes)
+* It is important to note that creating and deploying services and workloads on Kubernetes behaves in much the same way for Linux and Windows containers. [Kubectl commands](/docs/reference/kubectl/overview/) to interface with the cluster are identical. The example in the section below is provided simply to jumpstart your experience with Windows containers.
+
+## Getting Started: Deploying a Windows container
+
+To deploy a Windows container on Kubernetes, you must first create an example application. The example YAML file below creates a simple webserver application. Create a service spec named `win-webserver.yaml` with the contents below:
+
+```yaml
+ apiVersion: v1
+ kind: Service
+ metadata:
+ name: win-webserver
+ labels:
+ app: win-webserver
+ spec:
+ ports:
+ # the port that this service should serve on
+ - port: 80
+ targetPort: 80
+ selector:
+ app: win-webserver
+ type: NodePort
+ ---
+ apiVersion: extensions/v1beta1
+ kind: Deployment
+ metadata:
+ labels:
+ app: win-webserver
+ name: win-webserver
+ spec:
+ replicas: 2
+ template:
+ metadata:
+ labels:
+ app: win-webserver
+ name: win-webserver
+ spec:
+ containers:
+ - name: windowswebserver
+ image: mcr.microsoft.com/windows/servercore:ltsc2019
+ command:
+ - powershell.exe
+ - -command
+ - "<#code used from https://gist.github.com/wagnerandrade/5424431#> ; $$listener = New-Object System.Net.HttpListener ; $$listener.Prefixes.Add('http://*:80/') ; $$listener.Start() ; $$callerCounts = @{} ; Write-Host('Listening at http://*:80/') ; while ($$listener.IsListening) { ;$$context = $$listener.GetContext() ;$$requestUrl = $$context.Request.Url ;$$clientIP = $$context.Request.RemoteEndPoint.Address ;$$response = $$context.Response ;Write-Host '' ;Write-Host('> {0}' -f $$requestUrl) ; ;$$count = 1 ;$$k=$$callerCounts.Get_Item($$clientIP) ;if ($$k -ne $$null) { $$count += $$k } ;$$callerCounts.Set_Item($$clientIP, $$count) ;$$ip=(Get-NetAdapter | Get-NetIpAddress); $$header='Windows Container Web Server
' ;$$callerCountsString='' ;$$callerCounts.Keys | % { $$callerCountsString+='IP {0} callerCount {1} ' -f $$ip[1].IPAddress,$$callerCounts.Item($$_) } ;$$footer='' ;$$content='{0}{1}{2}' -f $$header,$$callerCountsString,$$footer ;Write-Output $$content ;$$buffer = [System.Text.Encoding]::UTF8.GetBytes($$content) ;$$response.ContentLength64 = $$buffer.Length ;$$response.OutputStream.Write($$buffer, 0, $$buffer.Length) ;$$response.Close() ;$$responseStatus = $$response.StatusCode ;Write-Host('< {0}' -f $$responseStatus) } ; "
+ nodeSelector:
+ beta.kubernetes.io/os: windows
```
-{
- "name": "net",
- "type": "ovn_cni.exe",
- "bridge": "br-int",
- "isGateway": "true",
- "ipMasq": "false",
- "ipam": {
- "type": "host-local",
- "subnet": "$SUBNET"
- }
-}
+
+{{< note >}}
+Port mapping is also supported, but for simplicity in this example the container port 80 is exposed directly to the service.
+{{< /note >}}
+
+1. Check that all nodes are healthy:
+
+ ```bash
+ kubectl get nodes
+ ```
+
+1. Deploy the service and watch for pod updates:
+
+ ```bash
+ kubectl apply -f win-webserver.yaml
+ kubectl get pods -o wide -w
+ ```
+
+ When the service is deployed correctly both Pods will be marked as Ready. To exit the watch command, press Ctrl+C.
+
+1. Check that the deployment succeeded. To verify:
+
+ * Two containers per pod on the Windows node, use `docker ps`
+ * Two pods listed from the Linux master, use `kubectl get pods`
+ * Node-to-pod communication across the network, `curl` port 80 of your pod IPs from the Linux master to check for a web server response
+ * Pod-to-pod communication, ping between pods (and across hosts, if you have more than one Windows node) using docker exec or kubectl exec
+ * Service-to-pod communication, `curl` the virtual service IP (seen under `kubectl get services`) from the Linux master and from individual pods
+ * Service discovery, `curl` the service name with the Kubernetes [default DNS suffix](/docs/concepts/services-networking/dns-pod-service/#services)
+ * Inbound connectivity, `curl` the NodePort from the Linux master or machines outside of the cluster
+ * Outbound connectivity, `curl` external IPs from inside the pod using kubectl exec
+
+{{< note >}}
+Windows container hosts are not able to access the IP of services scheduled on them due to current platform limitations of the Windows networking stack. Only Windows pods are able to access service IPs.
+{{< /note >}}
+
+## Managing Workload Identity with Group Managed Service Accounts
+
+Starting with Kubernetes v1.14, Windows container workloads can be configured to use Group Managed Service Accounts (GMSA). Group Managed Service Accounts are a specific type of Active Directory account that provides automatic password management, simplified service principal name (SPN) management, and the ability to delegate the management to other administrators across multiple servers. Containers configured with a GMSA can access external Active Directory Domain resources while carrying the identity configured with the GMSA. Learn more about configuring and using GMSA for Windows containers [here](/docs/tasks/configure-pod-container/configure-gmsa.md).
+
+## Taints and Tolerations
+
+Users today will need to use some combination of taints and node selectors in order to keep Linux and Windows workloads on their respective OS-specific nodes. This will likely impose a burden only on Windows users. The recommended approach is outlined below, with one of its main goals being that this approach should not break compatibility for existing Linux workloads.
+
+### Ensuring OS-specific workloads land on the appropriate container host
+
+Users can ensure Windows containers can be scheduled on the appropriate host using Taints and Tolerations. All Kubernetes nodes today have the following default labels:
+
+* beta.kubernetes.io/os = [windows|linux]
+* beta.kubernetes.io/arch = [amd64|arm64|...]
+
+If a Pod specification does not specify a nodeSelector like `"beta.kubernetes.io/os": windows`, it is possible the Pod can be scheduled on any host, Windows or Linux. This can be problematic since a Windows container can only run on Windows and a Linux container can only run on Linux. The best practice is to use a nodeSelector.
+
+However, we understand that in many cases users have a pre-existing large number of deployments for Linux containers, as well as an ecosystem of off-the-shelf configurations, such as community Helm charts, and programmatic Pod generation cases, such as with Operators. In those situations, you may be hesitant to make the configuration change to add nodeSelectors. The alternative is to use Taints. Because the kubelet can set Taints during registration, it could easily be modified to automatically add a taint when running on Windows only.
+
+For example: `--register-with-taints='os=Win1809:NoSchedule'`
+
+By adding a taint to all Windows nodes, nothing will be scheduled on them (that includes existing Linux Pods). In order for a Windows Pod to be scheduled on a Windows node, it would need both the nodeSelector to choose Windows, and the appropriate matching toleration.
+
+```yaml
+nodeSelector:
+ "beta.kubernetes.io/os": windows
+tolerations:
+ - key: "os"
+ operator: "Equal"
+ value: "Win1809"
+ effect: "NoSchedule"
```
-Where $SUBNET is the subnet that was used in the previous ```docker network create``` command.
-For a complete guide on Google Cloud Platform (GCP), namely Google Compute Engine (GCE) visit [this](https://github.com/apprenda/kubernetes-ovn-heterogeneous-cluster#heterogeneous-kubernetes-cluster-on-top-of-ovn).
+# Getting Help and Troubleshooting {#troubleshooting}
+
+Your main source of help for troubleshooting your Kubernetes cluster should start with this [section](/docs/tasks/debug-application-cluster/troubleshooting/). Some additional, Windows-specific troubleshooting help is included in this section. Logs are an important element of troubleshooting issues in Kubernetes. Make sure to include them any time you seek troubleshooting assistance from other contributors. Follow the instructions in the SIG-Windows [contributing guide on gathering logs](https://github.com/kubernetes/community/blob/master/sig-windows/CONTRIBUTING.md#gathering-logs).
+
+1. How do I know start.ps1 completed successfully?
+
+ You should see kubelet, kube-proxy, and (if you chose Flannel as your networking solution) flanneld host-agent processes running on your node, with running logs being displayed in separate PowerShell windows. In addition to this, your Windows node should be listed as "Ready" in your Kubernetes cluster.
+
+1. Can I configure the Kubernetes node processes to run in the background as services?
+
+ Kubelet and kube-proxy are already configured to run as native Windows Services, offering resiliency by re-starting the services automatically in the event of failure (for example a process crash). You have two options for configuring these node components as services.
+
+ 1. As native Windows Services
+
+ Kubelet & kube-proxy can be run as native Windows Services using `sc.exe`.
+
+ ```powershell
+ # Create the services for kubelet and kube-proxy in two separate commands
+ sc.exe create binPath= " --service "
+
+ # Please note that if the arguments contain spaces, they must be escaped.
+ sc.exe create kubelet binPath= "C:\kubelet.exe --service --hostname-override 'minion' "
+
+ # Start the services
+ Start-Service kubelet
+ Start-Service kube-proxy
-For a complete guide on Amazon Web Services (AWS) visit [this](https://github.com/justeat/kubernetes-windows-aws-ovs#kubernetes-on-windows-in-aws-using-ovn).
+ # Stop the service
+ Stop-Service kubelet (-Force)
+ Stop-Service kube-proxy (-Force)
+
+ # Query the service status
+ Get-Service kubelet
+ Get-Service kube-proxy
+ ```
-## Starting the Cluster
-To start your cluster, you'll need to start both the Linux-based Kubernetes control plane, and the Windows Server-based Kubernetes node components (kubelet and kube-proxy). For the OVS & OVN only the kubelet is required.
+ 1. Using nssm.exe
-## Starting the Linux-based Control Plane
-Use your preferred method to start Kubernetes cluster on Linux. Please note that Cluster CIDR might need to be updated.
+ You can also always use alternative service managers like [nssm.exe](https://nssm.cc/) to run these processes (flanneld, kubelet & kube-proxy) in the background for you. You can use this [sample script](https://github.com/Microsoft/SDN/tree/master/Kubernetes/flannel/register-svc.ps1), leveraging nssm.exe to register kubelet, kube-proxy, and flanneld.exe to run as Windows services in the background.
+
+ ```powershell
+ register-svc.ps1 -NetworkMode -ManagementIP -ClusterCIDR -KubeDnsServiceIP -LogDir
+
+ # NetworkMode = The network mode l2bridge (flannel host-gw, also the default value) or overlay (flannel vxlan) chosen as a network solution
+ # ManagementIP = The IP address assigned to the Windows node. You can use ipconfig to find this
+ # ClusterCIDR = The cluster subnet range. (Default value 10.244.0.0/16)
+ # KubeDnsServiceIP = The Kubernetes DNS service IP (Default value 10.96.0.10)
+ # LogDir = The directory where kubelet and kube-proxy logs are redirected into their respective output files (Default value C:\k)
+ ```
+
+ If the above referenced script is not suitable, you can manually configure nssm.exe using the following examples.
+ ```powershell
+ # Register flanneld.exe
+ nssm install flanneld C:\flannel\flanneld.exe
+ nssm set flanneld AppParameters --kubeconfig-file=c:\k\config --iface= --ip-masq=1 --kube-subnet-mgr=1
+ nssm set flanneld AppEnvironmentExtra NODE_NAME=
+ nssm set flanneld AppDirectory C:\flannel
+ nssm start flanneld
+
+ # Register kubelet.exe
+ nssm install kubelet C:\k\kubelet.exe
+ nssm set kubelet AppParameters --hostname-override= --v=6 --pod-infra-container-image=kubeletwin/pause --resolv-conf="" --allow-privileged=true --enable-debugging-handlers --cluster-dns= --cluster-domain=cluster.local --kubeconfig=c:\k\config --hairpin-mode=promiscuous-bridge --image-pull-progress-deadline=20m --cgroups-per-qos=false --log-dir= --logtostderr=false --enforce-node-allocatable="" --network-plugin=cni --cni-bin-dir=c:\k\cni --cni-conf-dir=c:\k\cni\config
+ nssm set kubelet AppDirectory C:\k
+ nssm start kubelet
+
+ # Register kube-proxy.exe (l2bridge / host-gw)
+ nssm install kube-proxy C:\k\kube-proxy.exe
+ nssm set kube-proxy AppDirectory c:\k
+ nssm set kube-proxy AppParameters --v=4 --proxy-mode=kernelspace --hostname-override=--kubeconfig=c:\k\config --enable-dsr=false --log-dir= --logtostderr=false
+ nssm.exe set kube-proxy AppEnvironmentExtra KUBE_NETWORK=cbr0
+ nssm set kube-proxy DependOnService kubelet
+ nssm start kube-proxy
+
+ # Register kube-proxy.exe (overlay / vxlan)
+ nssm install kube-proxy C:\k\kube-proxy.exe
+ nssm set kube-proxy AppDirectory c:\k
+ nssm set kube-proxy AppParameters --v=4 --proxy-mode=kernelspace --feature-gates="WinOverlay=true" --hostname-override= --kubeconfig=c:\k\config --network-name=vxlan0 --source-vip= --enable-dsr=false --log-dir= --logtostderr=false
+ nssm set kube-proxy DependOnService kubelet
+ nssm start kube-proxy
+ ```
+
+
+ For initial troubleshooting, you can use the following flags in [nssm.exe](https://nssm.cc/) to redirect stdout and stderr to a output file:
-## Support for kubeadm join
+ ```powershell
+ nssm set AppStdout C:\k\mysvc.log
+ nssm set AppStderr C:\k\mysvc.log
+ ```
-If your cluster has been created by [kubeadm](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/),
-and your networking is setup correctly using one of the methods listed above (networking is setup outside of kubeadm), you can use kubeadm to add a Windows node to your cluster. At a high level, you first have to initialize the master with kubeadm (Linux), then set up the CNI based networking (outside of kubeadm), and finally start joining Windows or Linux worker nodes to the cluster. For additional documentation and reference material, visit the kubeadm link above.
+ For additional details, see official [nssm usage](https://nssm.cc/usage) docs.
-The kubeadm binary can be found at [Kubernetes Releases](https://github.com/kubernetes/kubernetes/releases), inside the node binaries archive. Adding a Windows node is not any different than adding a Linux node:
+1. My Windows Pods do not have network connectivity
-`kubeadm.exe join --token : --discovery-token-ca-cert-hash sha256:`
+ If you are using virtual machines, ensure that MAC spoofing is enabled on all the VM network adapter(s).
-See [joining-your-nodes](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#joining-your-nodes) for more details.
+1. My Windows Pods cannot ping external resources
-## Supported Features
+ Windows Pods do not have outbound rules programmed for the ICMP protocol today. However, TCP/UDP is supported. When trying to demonstrate connectivity to resources outside of the cluster, please substitute `ping ` with corresponding `curl ` commands.
-The examples listed below assume running Windows nodes on Windows Server 1709. If you are running Windows Server 2016, the examples will need the image updated to specify `image: microsoft/windowsservercore:ltsc2016`. This is due to the requirement for container images to match the host operating system version when using process isolation. Not specifying a tag will implicitly use the `:latest` tag which can lead to surprising behaviors. Please consult with [https://hub.docker.com/r/microsoft/windowsservercore/](https://hub.docker.com/r/microsoft/windowsservercore/) for additional information on Windows Server Core image tagging.
+ If you are still facing problems, most likely your network configuration in [cni.conf](https://github.com/Microsoft/SDN/blob/master/Kubernetes/flannel/l2bridge/cni/config/cni.conf) deserves some extra attention. You can always edit this static file, the configuration will be applied to any newly created Kubernetes resources.
-### Scheduling Pods on Windows
-Because your cluster has both Linux and Windows nodes, you must explicitly set the `nodeSelector` constraint to be able to schedule pods to Windows nodes. You must set nodeSelector with the label `kubernetes.io/os` to the value `windows`; see the following example:
+ One of the Kubernetes networking requirements (see [Kubernetes model](/docs/concepts/cluster-administration/networking/)) is for cluster communication to occur without NAT internally. To honor this requirement, there is an [ExceptionList](https://github.com/Microsoft/SDN/blob/master/Kubernetes/flannel/l2bridge/cni/config/cni.conf#L20) for all the communication where we do not want outbound NAT to occur. However, this also means that you need to exclude the external IP you are trying to query from the ExceptionList. Only then will the traffic originating from your Windows pods be SNAT'ed correctly to receive a response from the outside world. In this regard, your ExceptionList in `cni.conf` should look as follows:
-{{< codenew file="windows/simple-pod.yaml" >}}
+ ```conf
+ "ExceptionList": [
+ "10.244.0.0/16", # Cluster subnet
+ "10.96.0.0/12", # Service subnet
+ "10.127.130.0/24" # Management (host) subnet
+ ]
+ ```
-{{< note >}}
-This example assumes you are running on Windows Server 1709, so uses the image tag to support that. If you are on a different version, you will need to update the tag. For example, if on Windows Server 2016, update to use `"image": "microsoft/iis"` which will default to that OS version.
-{{< /note >}}
+1. My Windows node cannot access NodePort service
-### Secrets and ConfigMaps
-Secrets and ConfigMaps can be utilized in Windows Server Containers, but must be used as environment variables. See limitations section below for additional details.
+ Local NodePort access from the node itself will fail. This is a known limitation. NodePort access will work from other nodes or external clients.
-**Examples:**
+1. vNICs and HNS endpoints of containers are being deleted
-Windows pod with secrets mapped to environment variables
+ This issue can be caused when the `hostname-override` parameter is not passed to [kube-proxy](/docs/reference/command-line-tools-reference/kube-proxy/). To resolve it, users need to pass the hostname to kube-proxy as follows:
-{{< codenew file="windows/secret-pod.yaml" >}}
+ ```powershell
+ C:\k\kube-proxy.exe --hostname-override=$(hostname)
+ ```
-Windows Pod with configMap values mapped to environment variables
+1. With flannel my nodes are having issues after rejoining a cluster
-{{< codenew file="windows/configmap-pod.yaml" >}}
+ Whenever a previously deleted node is being re-joined to the cluster, flannelD will try to assign a new pod subnet to the node. Users should remove the old pod subnet configuration files in the following paths:
-### Volumes
-Some supported Volume Mounts are local, emptyDir, hostPath. One thing to remember is that paths must either be escaped, or use forward slashes, for example `mountPath: "C:\\etc\\foo"` or `mountPath: "C:/etc/foo"`.
+ ```powershell
+ Remove-Item C:\k\SourceVip.json
+ Remove-Item C:\k\SourceVipRequest.json
+ ```
-Persistent Volume Claims are supported for supported volume types.
+1. After launching `start.ps1`, flanneld is stuck in "Waiting for the Network to be created"
-**Examples:**
+ There are numerous reports of this [issue which are being investigated](https://github.com/coreos/flannel/issues/1066); most likely it is a timing issue for when the management IP of the flannel network is set. A workaround is to simply relaunch start.ps1 or relaunch it manually as follows:
-Windows pod with a hostPath volume
+ ```powershell
+ PS C:> [Environment]::SetEnvironmentVariable("NODE_NAME", "")
+ PS C:> C:\flannel\flanneld.exe --kubeconfig-file=c:\k\config --iface= --ip-masq=1 --kube-subnet-mgr=1
+ ```
-{{< codenew file="windows/hostpath-volume-pod.yaml" >}}
+1. My Windows Pods cannot launch because of missing `/run/flannel/subnet.env`
-Windows pod with multiple emptyDir volumes
+ This indicates that Flannel didn't launch correctly. You can either try to restart flanneld.exe or you can copy the files over manually from `/run/flannel/subnet.env` on the Kubernetes master to` C:\run\flannel\subnet.env` on the Windows worker node and modify the `FLANNEL_SUBNET` row to a different number. For example, if node subnet 10.244.4.1/24 is desired:
-{{< codenew file="windows/emptydir-pod.yaml" >}}
+ ```env
+ FLANNEL_NETWORK=10.244.0.0/16
+ FLANNEL_SUBNET=10.244.4.1/24
+ FLANNEL_MTU=1500
+ FLANNEL_IPMASQ=true
+ ```
-### DaemonSets
+1. My Windows node cannot access my services using the service IP
-DaemonSets are supported
+ This is a known limitation of the current networking stack on Windows. Windows Pods are able to access the service IP however.
-{{< codenew file="windows/daemonset.yaml" >}}
+1. No network adapter is found when starting kubelet
-### Metrics
+ The Windows networking stack needs a virtual adapter for Kubernetes networking to work. If the following commands return no results (in an admin shell), virtual network creation — a necessary prerequisite for Kubelet to work — has failed:
-Windows Stats use a hybrid model: pod and container level stats come from CRI (via dockershim), while node level stats come from the "winstats" package that exports cadvisor like data structures using windows specific perf counters from the node.
+ ```powershell
+ Get-HnsNetwork | ? Name -ieq "cbr0"
+ Get-NetAdapter | ? Name -Like "vEthernet (Ethernet*"
+ ```
-### Container Resources
+ Often it is worthwhile to modify the [InterfaceName](https://github.com/Microsoft/SDN/blob/master/Kubernetes/flannel/l2bridge/start.ps1#L6) parameter of the start.ps1 script, in cases where the host's network adapter isn't "Ethernet". Otherwise, consult the output of the `start-kubelet.ps1` script to see if there are errors during virtual network creation.
-Container resources (CPU and memory) could be set now for windows containers in v1.10.
+1. My Pods are stuck at "Container Creating" or restarting over and over
-{{< codenew file="windows/deploy-resource.yaml" >}}
+ Check that your pause image is compatible with your OS version. The [instructions](https://docs.microsoft.com/en-us/virtualization/windowscontainers/kubernetes/deploying-resources) assume that both the OS and the containers are version 1803. If you have a later version of Windows, such as an Insider build, you will need to adjust the images accordingly. Please refer to the Microsoft's [Docker repository](https://hub.docker.com/u/microsoft/) for images. Regardless, both the pause image Dockerfile and the sample service will expect the image to be tagged as :latest.
-### Hyper-V Containers
+## Further investigation
-Hyper-V containers are supported as experimental in v1.10. To create a Hyper-V container, kubelet should be started with feature gates `HyperVContainer=true` and Pod should include annotation `experimental.windows.kubernetes.io/isolation-type=hyperv`.
+Check the DNS limitations for Windows in this [section](#dns-limitations).
-{{< codenew file="windows/deploy-hyperv.yaml" >}}
+If these steps don't resolve your problem, you can get help running Windows containers on Windows nodes in Kubernetes through:
-### Kubelet and kube-proxy can now run as Windows services
+* StackOverflow [Windows Server Container](https://stackoverflow.com/questions/tagged/windows-server-container) topic
+* Kubernetes Official Forum [discuss.kubernetes.io](https://discuss.kubernetes.io/)
+* Kubernetes Slack [#SIG-Windows Channel](https://kubernetes.slack.com/messages/sig-windows)
-Starting with kubernetes v1.11, kubelet and kube-proxy can run as Windows services.
+## Reporting Issues and Feature Requests
-This means that you can now register them as Windows services via `sc` command. More details about how to create Windows services with `sc` can be found [here](https://support.microsoft.com/en-us/help/251192/how-to-create-a-windows-service-by-using-sc-exe).
+If you have what looks like a bug, or you would like to make a feature request, please use the [Github issue tracking system](https://github.com/kubernetes/kubernetes/issues). You can open issues on [GitHub](https://github.com/kubernetes/kubernetes/issues/new/choose) and assign them to SIG-Windows. You should first search the list of issues in case it was reported previously and comment with your experience on the issue and add additional logs. SIG-Windows Slack is also a great avenue to get some initial support and troubleshooting ideas prior to creating a ticket.
-**Examples:**
+If filing a bug, please include detailed information about how to reproduce the problem, such as:
-To create the service:
-```
-PS > sc.exe create binPath= " --windows-service "
-CMD > sc create binPath= " --windows-service "
-```
-Please note that if the arguments contain spaces, it must be escaped. Example:
-```
-PS > sc.exe create kubelet binPath= "C:\kubelet.exe --windows-service --hostname-override 'minion' "
-CMD > sc create kubelet binPath= "C:\kubelet.exe --windows-service --hostname-override 'minion' "
-```
-To start the service:
-```
-PS > Start-Service kubelet; Start-Service kube-proxy
-CMD > net start kubelet && net start kube-proxy
-```
-To stop the service:
-```
-PS > Stop-Service kubelet (-Force); Stop-Service kube-proxy (-Force)
-CMD > net stop kubelet && net stop kube-proxy
-```
-To query the service:
-```
-PS > Get-Service kubelet; Get-Service kube-proxy;
-CMD > sc.exe queryex kubelet && sc qc kubelet && sc.exe queryex kube-proxy && sc.exe qc kube-proxy
-```
+* Kubernetes version: kubectl version
+* Environment details: Cloud provider, OS distro, networking choice and configuration, and Docker version
+* Detailed steps to reproduce the problem
+* [Relevant logs](https://github.com/kubernetes/community/blob/master/sig-windows/CONTRIBUTING.md#gathering-logs)
+* Tag the issue sig/windows by commenting on the issue with `/sig windows` to bring it to a SIG-Windows member's attention
+
+# Roadmap
+
+We have a lot of features in our roadmap. An abbreviated high level list is included below, but we encourage you to view our [roadmap project](https://github.com/orgs/kubernetes/projects/8) and help us make Windows support better by [contributing](https://github.com/kubernetes/community/blob/master/sig-windows/).
+
+## CRI-ContainerD
+
+ContainerD is another OCI-compliant runtime that recently graduated as a CNCF project. It's currently tested on Linux, but 1.3 will bring support for Windows and Hyper-V. [[reference](https://blog.docker.com/2019/02/containerd-graduates-within-the-cncf/)]
-## Known Limitations for Windows Server Containers with v1.9
+The CRI-ContainerD interface will be able to manage sandboxes based on Hyper-V. This provides a foundation where RuntimeClasses could be implemented for new use cases including:
-Some of these limitations will be addressed by the community in future releases of Kubernetes:
+* Hypervisor-based isolation between pods for additional security
+* Backwards compatibility allowing a node to run a newer Windows Server version without requiring containers to be rebuilt
+* Specific CPU/NUMA settings for a pod
+* Memory isolation and reservations
-- Shared network namespace (compartment) with multiple Windows Server containers (shared kernel) per pod is only supported on Windows Server 1709 or later
-- Using Secrets and ConfigMaps as volume mounts is not supported
-- Mount propagation is not supported on Windows
-- The StatefulSet functionality for stateful applications is not supported
-- Horizontal Pod Autoscaling for Windows Server Container pods has not been verified to work end-to-end
-- Hyper-V isolated containers are not supported.
-- Windows container OS must match the Host OS. If it does not, the pod will get stuck in a crash loop.
-- Under the networking models of L3 or Host GW, Kubernetes Services are inaccessible to Windows nodes due to a Windows issue. This is not an issue if using OVN/OVS for networking.
-- Windows kubelet.exe may fail to start when running on Windows Server under VMware Fusion [issue 57110](https://github.com/kubernetes/kubernetes/pull/57124)
-- Flannel and Weavenet are not yet supported
-- Some .Net Core applications expect environment variables with a colon (`:`) in the name. Kubernetes currently does not allow this. Replace colon (`:`) with double underscore (`__`) as documented [here](https://docs.microsoft.com/en-us/aspnet/core/fundamentals/configuration/?tabs=basicconfiguration#configuration-by-environment).
-- As cgroups are not supported on windows, kubelet.exe should be started with the following additional arguments `--cgroups-per-qos=false --enforce-node-allocatable=""` [issue 61716](https://github.com/kubernetes/kubernetes/issues/61716)
+## Deployment with kubeadm and cluster API
-## Next steps and resources
+Kubeadm is becoming the de facto standard for users to deploy a Kubernetes cluster. Windows node support in kubeadm will come in a future release. We are also making investments in cluster API to ensure Windows nodes are properly provisioned.
-- Support for Windows is in Beta as of v1.9 and your feedback is welcome. For information on getting involved, please head to [SIG-Windows](https://github.com/kubernetes/community/blob/master/sig-windows/README.md)
-- Troubleshooting and Common Problems: [Link](https://docs.microsoft.com/en-us/virtualization/windowscontainers/kubernetes/common-problems)
+## A few other big ticket items
+### Beta support for Group Managed Service Accounts
+### More CNIs
+### More Storage Plugins
diff --git a/content/en/docs/getting-started-guides/windows/flannel-master-kubeclt-get-pods.png b/content/en/docs/getting-started-guides/windows/flannel-master-kubeclt-get-pods.png
new file mode 100644
index 0000000000000..73da333fcfcaa
Binary files /dev/null and b/content/en/docs/getting-started-guides/windows/flannel-master-kubeclt-get-pods.png differ
diff --git a/content/en/docs/getting-started-guides/windows/flannel-master-kubectl-get-ds.png b/content/en/docs/getting-started-guides/windows/flannel-master-kubectl-get-ds.png
new file mode 100644
index 0000000000000..cda93533164ca
Binary files /dev/null and b/content/en/docs/getting-started-guides/windows/flannel-master-kubectl-get-ds.png differ
diff --git a/content/en/docs/getting-started-guides/windows/ovn_kubernetes.png b/content/en/docs/getting-started-guides/windows/ovn_kubernetes.png
deleted file mode 100644
index 739d75aad765c..0000000000000
Binary files a/content/en/docs/getting-started-guides/windows/ovn_kubernetes.png and /dev/null differ
diff --git a/content/en/docs/getting-started-guides/windows/windows-docker-error.png b/content/en/docs/getting-started-guides/windows/windows-docker-error.png
new file mode 100644
index 0000000000000..d00528c0d4cc4
Binary files /dev/null and b/content/en/docs/getting-started-guides/windows/windows-docker-error.png differ
diff --git a/content/en/docs/getting-started-guides/windows/windows-setup.png b/content/en/docs/getting-started-guides/windows/windows-setup.png
deleted file mode 100644
index e11c58d596e35..0000000000000
Binary files a/content/en/docs/getting-started-guides/windows/windows-setup.png and /dev/null differ