-
Notifications
You must be signed in to change notification settings - Fork 364
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add containerd runtime support for antrea agent on windows #4279
Merged
Merged
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,279 @@ | ||
| apiVersion: v1 | ||
| data: | ||
| Install-WindowsCNI-Containerd.ps1: | | ||
| $ErrorActionPreference = "Stop"; | ||
| mkdir -force c:/var/log/antrea | ||
|
|
||
| $mountPath = $env:CONTAINER_SANDBOX_MOUNT_POINT | ||
| $mountPath = ($mountPath.Replace('\', '/')).TrimEnd('/') | ||
| mkdir -force C:/var/run/secrets/kubernetes.io/serviceaccount | ||
| cp $mountPath/var/run/secrets/kubernetes.io/serviceaccount/ca.crt C:/var/run/secrets/kubernetes.io/serviceaccount | ||
| cp $mountPath/var/run/secrets/kubernetes.io/serviceaccount/token C:/var/run/secrets/kubernetes.io/serviceaccount | ||
| cp $mountPath/k/antrea/cni/* c:/opt/cni/bin/ | ||
| cp $mountPath/etc/antrea/antrea-cni.conflist c:/etc/cni/net.d/10-antrea.conflist | ||
| mkdir -force c:/k/antrea/bin | ||
| cp $mountPath/k/antrea/bin/antctl.exe c:/k/antrea/bin/antctl.exe | ||
|
|
||
| Run-AntreaAgent-Containerd.ps1: | | ||
| $ErrorActionPreference = "Stop" | ||
| $mountPath = $env:CONTAINER_SANDBOX_MOUNT_POINT | ||
| $mountPath = ($mountPath.Replace('\', '/')).TrimEnd('/') | ||
| & "$mountPath/k/antrea/bin/antrea-agent.exe" --config=$mountPath/etc/antrea/antrea-agent.conf --logtostderr=false --log_dir=c:/var/log/antrea --alsologtostderr --log_file_max_size=100 --log_file_max_num=4 --v=4 | ||
| kind: ConfigMap | ||
| metadata: | ||
| labels: | ||
| app: antrea | ||
| name: antrea-agent-windows-kht6m7hthm | ||
| namespace: kube-system | ||
| --- | ||
| apiVersion: v1 | ||
| data: | ||
| antrea-agent.conf: | | ||
| # FeatureGates is a map of feature names to bools that enable or disable experimental features. | ||
| featureGates: | ||
| # Enable antrea proxy which provides ServiceLB for in-cluster services in antrea agent. | ||
| # It should be enabled on Windows, otherwise NetworkPolicy will not take effect on | ||
| # Service traffic. | ||
| # AntreaProxy: true | ||
|
|
||
| # Enable EndpointSlice support in AntreaProxy. Don't enable this feature unless that EndpointSlice | ||
| # API version v1beta1 is supported and set as enabled in Kubernetes. If AntreaProxy is not enabled, | ||
| # this flag will not take effect. | ||
| # EndpointSlice: false | ||
|
|
||
| # Enable NodePortLocal feature to make the Pods reachable externally through NodePort | ||
| # NodePortLocal: true | ||
|
|
||
| # Enable flowexporter which exports polled conntrack connections as IPFIX flow records from each agent to a configured collector. | ||
| # FlowExporter: false | ||
|
|
||
| # Name of the OpenVSwitch bridge antrea-agent will create and use. | ||
| # Make sure it doesn't conflict with your existing OpenVSwitch bridges. | ||
| #ovsBridge: br-int | ||
|
|
||
| # Name of the interface antrea-agent will create and use for host <--> pod communication. | ||
| # Make sure it doesn't conflict with your existing interfaces. | ||
| #hostGateway: antrea-gw0 | ||
|
|
||
| # Encapsulation mode for communication between Pods across Nodes, supported values: | ||
| # - geneve (default) | ||
| # - vxlan | ||
| # - stt | ||
| #tunnelType: geneve | ||
|
|
||
| # TunnelPort is the destination port for UDP and TCP based tunnel protocols | ||
| # (Geneve, VXLAN, and STT). If zero, it will use the assigned IANA port for the | ||
| # protocol, i.e. 6081 for Geneve, 4789 for VXLAN, and 7471 for STT. | ||
| #tunnelPort: 0 | ||
|
|
||
| # Default MTU to use for the host gateway interface and the network interface of each Pod. | ||
| # If omitted, antrea-agent will discover the MTU of the Node's primary interface and | ||
| # also adjust MTU to accommodate for tunnel encapsulation overhead. | ||
| #defaultMTU: 1450 | ||
|
|
||
| # ClusterIP CIDR range for Services. It's required when AntreaProxy is not enabled, and should be | ||
| # set to the same value as the one specified by --service-cluster-ip-range for kube-apiserver. When | ||
| # AntreaProxy is enabled, this parameter is not needed and will be ignored if provided. | ||
| #serviceCIDR: 10.96.0.0/12 | ||
|
|
||
| # The port for the antrea-agent APIServer to serve on. | ||
| #apiPort: 10350 | ||
|
|
||
| # Enable metrics exposure via Prometheus. Initializes Prometheus metrics listener. | ||
| #enablePrometheusMetrics: true | ||
|
|
||
| # Provide the IPFIX collector address as a string with format <HOST>:[<PORT>][:<PROTO>]. | ||
| # HOST can only be IP right now because there is a DNS resolution issue in current Windows support. | ||
| # IP can be either IPv4 or IPv6. However, IPv6 address should be wrapped with []. | ||
| # If PORT is empty, we default to 4739, the standard IPFIX port. | ||
| # If no PROTO is given, we consider "tls" as default. We support "tls", "tcp" and | ||
| # "udp" protocols. "tls" is used for securing communication between flow exporter and | ||
| # flow aggregator. | ||
| #flowCollectorAddr: "flow-aggregator.flow-aggregator.svc:4739:tls" | ||
|
|
||
| # Provide flow poll interval as a duration string. This determines how often the | ||
| # flow exporter dumps connections from the conntrack module. Flow poll interval | ||
| # should be greater than or equal to 1s (one second). | ||
| # Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h". | ||
| #flowPollInterval: "5s" | ||
|
|
||
| # Provide the active flow export timeout, which is the timeout after which a flow | ||
| # record is sent to the collector for active flows. Thus, for flows with a continuous | ||
| # stream of packets, a flow record will be exported to the collector once the elapsed | ||
| # time since the last export event is equal to the value of this timeout. | ||
| # Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h". | ||
| #activeFlowExportTimeout: "30s" | ||
|
|
||
| # Provide the idle flow export timeout, which is the timeout after which a flow | ||
| # record is sent to the collector for idle flows. A flow is considered idle if no | ||
| # packet matching this flow has been observed since the last export event. | ||
| # Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h". | ||
| #idleFlowExportTimeout: "15s" | ||
|
|
||
| # Enable TLS communication from flow exporter to flow aggregator. | ||
| #enableTLSToFlowAggregator: true | ||
|
|
||
| # Determines how traffic is encapsulated. It has the following options: | ||
| # encap(default): Inter-node Pod traffic is always encapsulated and Pod to external network | ||
| # traffic is SNAT'd. | ||
| # noEncap: Inter-node Pod traffic is not encapsulated; Pod to external network traffic is | ||
| # SNAT'd if noSNAT is not set to true. Underlying network must be capable of | ||
| # supporting Pod traffic across IP subnets. | ||
| # hybrid: noEncap if source and destination Nodes are on the same subnet, otherwise encap. | ||
| # | ||
| #trafficEncapMode: encap | ||
|
|
||
| # The name of the interface on Node which is used for tunneling or routing the traffic across Nodes. | ||
| # If there are multiple IP addresses configured on the interface, the first one is used. The IP | ||
| # address used for tunneling or routing traffic to remote Nodes is decided in the following order of | ||
| # preference (from highest to lowest): | ||
| # 1. transportInterface | ||
| # 2. transportInterfaceCIDRs | ||
| # 3. The Node IP | ||
| #transportInterface: | ||
|
|
||
| # The network CIDRs of the interface on Node which is used for tunneling or routing the traffic across | ||
| # Nodes. If there are multiple interfaces configured the same network CIDR, the first one is used. The | ||
| # IP address used for tunneling or routing traffic to remote Nodes is decided in the following order of | ||
| # preference (from highest to lowest): | ||
| # 1. transportInterface | ||
| # 2. transportInterfaceCIDRs | ||
| # 3. The Node IP | ||
| #transportInterfaceCIDRs: [<IPv4 CIDR>,<IPv6 CIDR>] | ||
|
|
||
| # Provide the address of Kubernetes apiserver, to override any value provided in kubeconfig or InClusterConfig. | ||
| # Defaults to "". It must be a host string, a host:port pair, or a URL to the base of the apiserver. | ||
| #kubeAPIServerOverride: "" | ||
|
|
||
| # Option antreaProxy contains AntreaProxy related configuration options. | ||
| antreaProxy: | ||
| # ProxyAll tells antrea-agent to proxy ClusterIP Service traffic, regardless of where they come from. | ||
| # Therefore, running kube-proxy is no longer required. This requires the AntreaProxy feature to be enabled. | ||
| # Note that this option is experimental. If kube-proxy is removed, option kubeAPIServerOverride must be used to access | ||
| # apiserver directly. | ||
| #proxyAll: false | ||
|
|
||
| nodePortLocal: | ||
| # Enable NodePortLocal, a feature used to make Pods reachable using port forwarding on the host. To | ||
| # enable this feature, you need to set "enable" to true, and ensure that the NodePortLocal feature | ||
| # gate is also enabled (which is the default). | ||
| # enable: false | ||
| # Provide the port range used by NodePortLocal. When the NodePortLocal feature is enabled, a port | ||
| # from that range will be assigned whenever a Pod's container defines a specific port to be exposed | ||
| # (each container can define a list of ports as pod.spec.containers[].ports), and all Node traffic | ||
| # directed to that port will be forwarded to the Pod. | ||
| # portRange: 61000-62000 | ||
| antrea-cni.conflist: | | ||
| { | ||
| "cniVersion":"0.3.0", | ||
| "name": "antrea", | ||
| "plugins": [ | ||
| { | ||
| "type": "antrea", | ||
| "ipam": { | ||
| "type": "host-local" | ||
| }, | ||
| "capabilities": {"dns": true} | ||
| } | ||
| ] | ||
| } | ||
| kind: ConfigMap | ||
| metadata: | ||
| labels: | ||
| app: antrea | ||
| name: antrea-windows-config-8kfkb8t957 | ||
| namespace: kube-system | ||
| --- | ||
| apiVersion: apps/v1 | ||
| kind: DaemonSet | ||
| metadata: | ||
| labels: | ||
| app: antrea | ||
| component: antrea-agent | ||
| name: antrea-agent-windows | ||
| namespace: kube-system | ||
| spec: | ||
| selector: | ||
| matchLabels: | ||
| app: antrea | ||
| component: antrea-agent | ||
| template: | ||
| metadata: | ||
| annotations: | ||
| "microsoft.com/hostprocess-inherit-user": "true" | ||
| labels: | ||
| app: antrea | ||
| component: antrea-agent | ||
| spec: | ||
| securityContext: | ||
| windowsOptions: | ||
| runAsUserName: "NT AUTHORITY\\SYSTEM" | ||
| hostProcess: true | ||
| containers: | ||
| - args: | ||
| - -file | ||
| - $env:CONTAINER_SANDBOX_MOUNT_POINT/var/lib/antrea-windows/Run-AntreaAgent-Containerd.ps1 | ||
| command: | ||
| - powershell | ||
| env: | ||
| - name: POD_NAME | ||
| valueFrom: | ||
| fieldRef: | ||
| fieldPath: metadata.name | ||
| - name: POD_NAMESPACE | ||
| valueFrom: | ||
| fieldRef: | ||
| fieldPath: metadata.namespace | ||
| - name: NODE_NAME | ||
| valueFrom: | ||
| fieldRef: | ||
| fieldPath: spec.nodeName | ||
| image: antrea/antrea-windows:latest | ||
| imagePullPolicy: IfNotPresent | ||
| name: antrea-agent | ||
| volumeMounts: | ||
| - mountPath: /etc/antrea | ||
| name: antrea-windows-config | ||
| - mountPath: /var/lib/antrea-windows | ||
| name: antrea-agent-windows | ||
| - mountPath: /var/log/antrea/ | ||
| name: var-log-antrea | ||
| hostNetwork: true | ||
| initContainers: | ||
| - args: | ||
| - -File | ||
| - $env:CONTAINER_SANDBOX_MOUNT_POINT/var/lib/antrea-windows/Install-WindowsCNI-Containerd.ps1 | ||
| command: | ||
| - powershell | ||
| image: antrea/antrea-windows:latest | ||
| imagePullPolicy: IfNotPresent | ||
| name: install-cni | ||
| volumeMounts: | ||
| - mountPath: /etc/antrea | ||
| name: antrea-windows-config | ||
| readOnly: true | ||
| - mountPath: /var/lib/antrea-windows | ||
| name: antrea-agent-windows | ||
| nodeSelector: | ||
| kubernetes.io/os: windows | ||
| priorityClassName: system-node-critical | ||
| serviceAccountName: antrea-agent | ||
| tolerations: | ||
| - key: CriticalAddonsOnly | ||
| operator: Exists | ||
| - effect: NoSchedule | ||
| operator: Exists | ||
| volumes: | ||
| - configMap: | ||
| name: antrea-windows-config-8kfkb8t957 | ||
| name: antrea-windows-config | ||
| - configMap: | ||
| defaultMode: 420 | ||
| name: antrea-agent-windows-kht6m7hthm | ||
| name: antrea-agent-windows | ||
| - hostPath: | ||
| path: /var/log/antrea/ | ||
| type: DirectoryOrCreate | ||
| name: var-log-antrea | ||
| updateStrategy: | ||
| type: RollingUpdate | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,88 @@ | ||
| apiVersion: apps/v1 | ||
| kind: DaemonSet | ||
| metadata: | ||
| labels: | ||
| component: antrea-agent | ||
| name: antrea-agent-windows | ||
| namespace: kube-system | ||
| spec: | ||
| selector: | ||
| matchLabels: | ||
| component: antrea-agent | ||
| template: | ||
| metadata: | ||
| annotations: | ||
| "microsoft.com/hostprocess-inherit-user": "true" | ||
| labels: | ||
| component: antrea-agent | ||
| spec: | ||
| securityContext: | ||
| windowsOptions: | ||
| runAsUserName: "NT AUTHORITY\\SYSTEM" | ||
| hostProcess: true | ||
| containers: | ||
| - args: | ||
| - -file | ||
| - $env:CONTAINER_SANDBOX_MOUNT_POINT/var/lib/antrea-windows/Run-AntreaAgent-Containerd.ps1 | ||
| command: | ||
| - powershell | ||
| env: | ||
| - name: POD_NAME | ||
| valueFrom: | ||
| fieldRef: | ||
| fieldPath: metadata.name | ||
| - name: POD_NAMESPACE | ||
| valueFrom: | ||
| fieldRef: | ||
| fieldPath: metadata.namespace | ||
| - name: NODE_NAME | ||
| valueFrom: | ||
| fieldRef: | ||
| fieldPath: spec.nodeName | ||
| image: antrea-windows | ||
| name: antrea-agent | ||
| volumeMounts: | ||
| - mountPath: /etc/antrea | ||
| name: antrea-windows-config | ||
| - mountPath: /var/lib/antrea-windows | ||
| name: antrea-agent-windows | ||
| - mountPath: /var/log/antrea/ | ||
| name: var-log-antrea | ||
| hostNetwork: true | ||
| initContainers: | ||
| - args: | ||
| - -File | ||
| - $env:CONTAINER_SANDBOX_MOUNT_POINT/var/lib/antrea-windows/Install-WindowsCNI-Containerd.ps1 | ||
| command: | ||
| - powershell | ||
| image: antrea-windows | ||
| name: install-cni | ||
| volumeMounts: | ||
| - mountPath: /etc/antrea | ||
| name: antrea-windows-config | ||
| readOnly: true | ||
| - mountPath: /var/lib/antrea-windows | ||
| name: antrea-agent-windows | ||
| nodeSelector: | ||
| kubernetes.io/os: windows | ||
| priorityClassName: system-node-critical | ||
| serviceAccountName: antrea-agent | ||
| tolerations: | ||
| - key: CriticalAddonsOnly | ||
| operator: Exists | ||
| - effect: NoSchedule | ||
| operator: Exists | ||
| volumes: | ||
| - configMap: | ||
| name: antrea-windows-config | ||
| name: antrea-windows-config | ||
| - configMap: | ||
| defaultMode: 420 | ||
| name: antrea-agent-windows | ||
| name: antrea-agent-windows | ||
| - hostPath: | ||
| path: /var/log/antrea/ | ||
| type: DirectoryOrCreate | ||
| name: var-log-antrea | ||
| updateStrategy: | ||
| type: RollingUpdate |
11 changes: 11 additions & 0 deletions
11
build/yamls/windows/base/conf/Install-WindowsCNI-Containerd.ps1
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| $ErrorActionPreference = "Stop"; | ||
| mkdir -force c:/var/log/antrea | ||
| $mountPath = $env:CONTAINER_SANDBOX_MOUNT_POINT | ||
| $mountPath = ($mountPath.Replace('\', '/')).TrimEnd('/') | ||
| mkdir -force C:/var/run/secrets/kubernetes.io/serviceaccount | ||
| cp $mountPath/var/run/secrets/kubernetes.io/serviceaccount/ca.crt C:/var/run/secrets/kubernetes.io/serviceaccount | ||
| cp $mountPath/var/run/secrets/kubernetes.io/serviceaccount/token C:/var/run/secrets/kubernetes.io/serviceaccount | ||
| cp $mountPath/k/antrea/cni/* c:/opt/cni/bin/ | ||
| cp $mountPath/etc/antrea/antrea-cni.conflist c:/etc/cni/net.d/10-antrea.conflist | ||
| mkdir -force c:/k/antrea/bin | ||
| cp $mountPath/k/antrea/bin/antctl.exe c:/k/antrea/bin/antctl.exe |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| $ErrorActionPreference = "Stop" | ||
| $mountPath = $env:CONTAINER_SANDBOX_MOUNT_POINT | ||
| $mountPath = ($mountPath.Replace('\', '/')).TrimEnd('/') | ||
| & "$mountPath/k/antrea/bin/antrea-agent.exe" --config=$mountPath/etc/antrea/antrea-agent.conf --logtostderr=false --log_dir=c:/var/log/antrea --alsologtostderr --log_file_max_size=100 --log_file_max_num=4 --v=4 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You have copied the files from
$mountPathtoc:\k\antrea, but used the file in $mountPath when running antrea-agent, I thought you wanted to run the file in c:/k ? Otherwise, you don't need to copy the files.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we only need antctl.exe in C:\k\antrea, the reason is we need the same path in e2e test.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I didn't get your point, firstly we want to run
C:\k\antrea\antctl.exeinside Windows Pod or on Windows Node host?? 2) Why can not copy the file in initContainers?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In function waitForNetworkpolicyRealized we have "antctlCmd := fmt.Sprintf("C:/k/antrea/bin/antctl.exe get networkpolicy -S %s -n %s -T %s", networkpolicy, data.testNamespace, npOption)". If we don't copy the antctl.exe, the traceflow command will return 'file not exist' error.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the comments, I will add another patch to support different antctl.exe path for docker and containerd.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since we already support running antrea-agent inside container not on the host, I don't think t is needed to run antctl on the host any more. This is unlike docker, in which case antrea-agent is actually running on the host, which requires antctl to run on the host to ensure the connection to antctl is working as expected.