Support Multicast statistics

[ceclinux]

Describe what you are trying to solve

This proposal is to collect and expose the statistical data of multicast. The requirements are listed below:

Per Pods stats

Per Pod interface counter for:
1. Multicast packet successfully transmitted by the Pod interface(per node)
2. Multicast packet successfully received by the Pod interface(per node)

Networkpolicy stats for

3. Multicast packet allowed by on transmit by the Pod interface(network policy and cluster-wide)
4. Multicast packet dropped by on transmit by the Pod interface(network policy and cluster-wide)
5. IGMP/MLD packet discarded(network policy and cluster-wide)
6. IGMP/MLD packet accepted and acted upon by Antrea(network policy and cluster-wide)

Per multicast group stats

Showing the list of Pod interfaces that:
7. Registered dynamically (IGMP/MLD) an interest in a specific multicast group(cluster-wide)

One kubectl command and one antctl command are introduced to display these statistics. The sample command outputs would be:

antctl get podmulticaststats

POD                      INBOUND             OUTBOUND
namespace/mcjoin-k95gb   3394                232                    

The command implements the first two requirements of Per Pods stats. It is running in Antrea-agent and shows the Pod statistics of multicast traffic at node level.

kubectl get multicastgroups

GROUP         PODS
224.3.4.5     default/mcjoin-k95gb, test/mcjoin-aabbc

The command implements the requirements 7 for Per multicast group stats.

The requirements 3,4,5,6 will be implemented with NetworkpolicyStats module in Antrea, users can get Multicast or IGMP/MLD statistics allowed/dropped by querying kubectl get antreaclusternetworkpolicystats -o json and find the specific statistics with ruleName of Networkpolicy that applied. Please check https://github.com/antrea-io/antrea/blob/main/docs/feature-gates.md#networkpolicystats for sample commands outputs.

APIs

Internal

type MulticastGroupInfo struct {
	// Group is the IP of the multicast group.
	Group string
	// Pods is the list of Pods that have joined the multicast group.
	Pods []PodReference
}


// Add field PodMulticastStats to the original NodeStatsSummary from pkg/apis/controlplane/types.go
type NodeStatsSummary struct {
	....
        // Multicast group information from the Node.
	Multicast []MulticastGroupInfo
}

There is already a defined struct called NodeStatsSummary which contains stats produced on a Node. It's used by the Antrea-agents to report stats to the Antrea-controller. We add an extra field called Multicast which contains mapping of multicast group to Pods for each node. The calculation of mapping is achieved by PacketIn of IGMP report packets, which will be discussed below.

The report of Multicast is not incremental. Antrea-controller could calculate the group Pods mapping in the whole cluster and get exposed by kubectl get multicastgroups. For instance, in node A, Antrea-agent reports 224.3.4.5 with Pods default/PodA, default/PodB, and in node B, Antrea-agent reports 224.2.3.4 with default/PodA. Antrea-controller will update the mapping, the command gives

kubectl get multicastgroups

GROUP         PODS
224.3.4.5     default/PodA, default/PodB
224.2.3.4     default/PodA

After next report cycle, in node A, Antrea-agent reports 224.3.4.5 with default/PodA, default/PodC, and in node B, Antrea-agent reports 224.2.3.4 with empty Pods array. Antrea-controller will refresh the mapping, the command gives

GROUP         PODS
224.3.4.5     default/PodA, default/PodC

Also, we defined an antctl endpoint with podmulticaststats api

	{
			use:   "podmulticaststats",
			short: "Show multicast statistics",
			long:  "Show multicast traffic statistics of Pods",
			example: `  Show multicast traffic statistics of all local Pods on the Node
$ antctl get podmulticaststats
Show multicast traffic statistics of a given Pod
$ antctl get podmulticaststats pod -n namespace`,
			commandGroup: get,
			agentEndpoint: &endpoint{
				nonResourceEndpoint: &nonResourceEndpoint{
					path:       "/podmulticaststats",
					outputType: multiple,
					params: []flagInfo{
						{
							name:  "name",
							usage: "Retrieve Pod Multicast Statistics by name. If present, Namespace must be provided.",
							arg:   true,
						},
						{
							name:      "namespace",
							usage:     "Get Pod Multicast Statistics from specific Namespace.",
							shorthand: "n",
						},
					},
				},
			},

			transformedResponse: reflect.TypeOf(multicast.Response{}),
		},

Public API

The API must follow the K8s convention so that it can be registered as an APIService and accessed by kubectl get multicastgroups.

Antrea-controller will aggregate the stats reported in NodeStatsSummary by Antrea-agent and expose it to kubectl

// PodReference represents a Pod Reference.
type PodReference struct {
	// The name of this Pod.
	Name string
	// The namespace of this Pod.
	Namespace string
}

// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

// MulticastGroup contains the mapping between multicast group and Pods.
type MulticastGroup struct {
	metav1.TypeMeta
	metav1.ObjectMeta

	// Group is the IP of the multicast group.
	Group string
	// Pods is the list of Pods that have joined the multicast group.
	Pods []PodReference
}

// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

// MulticastGroupList is a list of MulticastGroup.
type MulticastGroupList struct {
	metav1.TypeMeta
	metav1.ListMeta

	// List of MulticastGroup.
	Items []MulticastGroup
}

The API group is stats.antrea.io , and the endpoints are /apis/stats.antrea.io/v1alpha1/podmulticaststats and /apis/stats.antrea.io/v1alpha1/multicastgroups.

How to get these stats

Multicast is retrieved by processing packetIn of IGMP report message. The packetIn message contains Pod info and multicast IP that wants to join.

After parsing the packetIn message in antrea-agent, func validate(event, igmpType, packetInData) is called, which will return allowed or dropped action enforced by Networkpolicy for the Pod and specified multicast address. Then Antrea-agent will choose to continue processing the IGMP report message or drop it. Meanwhile, the statistical data of IGMP report packets that acted by Networkpolicy is calculated as well(IGMP report of requirement 5 and 6).

For non-IGMP multicast traffic statistics acted by Networkpolicy, the data comes from flows that are created by Antrea-agents, and will be process by Networkpolicystats pipeline that has already created by Antrea. See #3660 and #3323. Flow examples:

// example MulticastEgressMetric allow flow format(example 3):
// table=MulticastEgressMetric, n_packets=11, n_bytes=1562, priority=200,reg0=0x400/0x400,reg3=0x4 actions=goto_table:MulticastEgressPodMetric
// example MulticastEgressMetric drop flow format(example 4):
// table=MulticastEgressMetric, n_packets=230, n_bytes=32660, priority=200,reg0=0x400/0x400,reg3=0x3 actions=drop
// example MulticastIngressMetric allow flow format(IGMP query of requirements 5 and 6):
// table=MulticastIngressMetric, n_packets=18, n_bytes=780, priority=200,reg0=0x400/0x400,reg3=0x3 actions=resubmit(,MulticastOutput)

The data that feeds antctl get podmulticaststats api is from the flows that update when Pod created or deleted. When pod is added by cluster, one flow entry will be created on MulticastEgressPodMetric and MulticastIngressPodMetric, respectively, with their IP(nw_src) and port(reg1) to match inbound multicast and outbound multicast statistics for this Pod.

// Pod inbound multicast traffic
table=MulticastIngressPodMetric, n_packets=0, n_bytes=0, priority=200,ip,reg1=0x5 actions=goto_table:MulticastOutput
// Pod Outbound multicast traffic
table=MulticastEgressPodMetric, n_packets=0, n_bytes=0, priority=200,ip,nw_src=10.244.0.5 actions=goto_table:MulticastRouting

Everytime users try to query antctl get podmulticaststats, the flows will be parsed and displayed as an example described above.

Prometheus Integration
Not plan to integrate it as discussed in Support NetworkPolicy statistics(#985)

Additional context
PRs for this feature:

• • [Multicast] Add Multicast statistics API #3354
• • Multicast statistics implementation #3449

[ceclinux]

Update:

Networkpolicy stats for

3. Multicast packet allowed by on transmit by the Pod interface(network policy and cluster-wide)
4. Multicast packet dropped by on transmit by the Pod interface(network policy and cluster-wide)
5. IGMP/MLD packet discarded(network policy and cluster-wide)
6. IGMP/MLD packet accepted and acted upon by Antrea(network policy and cluster-wide)

The command kubectl get podmulticaststats is removed from the design as the per pod statistics for inbound_dropped, outbound_dropped, igmp_allowd, igmp_dropped are removed from the requirement. These counters are only needed to be shown networkpolicy level.

In addition, numbers like inbound_dropped, outbound_dropped, igmp_allowd, igmp_dropped can also be obtained by kubectl networkpolicystats -o json, which shows rule stats per networkpolicy. The rules stats for multicast networkpolicy can serve to differentiate one metric from other multicast metrics because only one kind of multicast traffic can be blocked or allowed from a single NP rule.

The deleted command:

kubectl get podmulticaststats

POD          INBOUND_DROPPED   OUTBOUND_DROPPED   IGMP_ALLOWED	IGMP_DROPPED
mcjoin-k95gb 2233              0                  55            22

kubectl get podmulticaststats -o json

{
    "apiVersion": "v1",
    "items": [
        {
            "apiVersion": "stats.antrea.io/v1alpha1",
            "kind": "PodMulticastStats",
            metadata: {
                    name: mcjoin-k95gb,
                    namespace: default,
                    .....
            }
            "networkPolicyTrafficStats": [
                {
                    "networkpolicy": {
                       Type: "AntreaNetworkPolicy",
                       Name: "multicast_np",
                       Namespace: "test",  
                    },
                    "trafficStats": {
                        "inbound_dropped": 392,
                        "outbound_dropped": 4,
                        "igmp_allowd": 2,
                        "igmp_dropped": 22,
                    }
                }
            ],
            "trafficStats": {
               "inbound_dropped": 392,
               "outbound_dropped": 4,
               "igmp_allowd": 2,
               "igmp_dropped": 22,
            }
        },
    ],
}

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment, or this will be closed in 90 days