Support applying ClusterNetworkPolicy to Nodes in Antrea

[tnqn]

Describe what you are trying to solve

#4213 asked for the ability to apply NetworkPolicies to not only Pods but also Kubernetes Nodes. Antrea currently has a feature called ExternalNode, which enables users to apply NetworkPolicies for non-Kubernetes Nodes. The two requirements are kind of the same thing from a technical perspective.

The implementation of ExternalNode moves the primary network interfaces of Nodes to OVS bridge and uses OpenFlow flows to enforce policies, which met great chanllenges in compatibility with OSes, especially when DHCP is used and on cloud platforms. #5192 and #5221 are two examples.

This proposal describes a simple and steady way to apply NetworkPolicies to Kubernetes Nodes, which can also be used by ExternalNode in the future, unifying the implementation and resolving the known problems.

Describe the solution you have in mind

Dataplane

There could be four technical approaches to apply NetworkPolicies to Nodes:

Approach	Pros	Cons
OVS, bridge network interface	1. Maximum code reuse 2. Effcient	1. Hard to support multiple interfaces 2. Hard to be compaible with various network device managers
tc-eBPF	1. Have full control on packets	1. Hard to support multiple interfaces 2. More efforts to implement firewall from scratch: including priority-based packet classier, connection tracking
OVS with tc-eBPF redirecting packets only	1. Good code reuse	1. Hard to support multiple interfaces 2. Complex datapath 3. Uncertain side effect
iptables/netfilter	1. Mature firewall mechanism 2. Easy to support multiple interfaces	1. Poor performance under a significant number of rule

All things considered, iptables/netfilter seems the most practicable approach if its performance disadvantage doesn't cause a problem in this use case. Then let's look at how it behaves with different numbers of rules. I have ran benchmark using iperf and netperf with different number rules in iptables filter table and here are the numbers:

As seen from the numbers, throughput (by Bitrate) and latency (by TCP_RR) are almost unaffected by the number of rules, which meets the expectation as the number of rules only affects the first packet of a connection in most cases. TCP_CRR became worse while number of rules increased. It decreased 9.09% with 50 rules and 13.6% with 100 rules. But considering the use case and the expectation that one NetworkPolicy rule will be mapped to an iptables rule regardless of the number of IPs used by it (thanks to ipset), it's unlikely that the number of rules applied to a single Node could exceed 100 and perhaps could be less as users will likely just deny all and only allow specific accesses on the principle of least privilege.

So we propose to use iptables/netfilter in dataplane.

API Change

Considring Node is a cluster scope resource, and the current API types in Antrea, we will make the following API changes:

1. A policy/rule's AppliedTo can use NodeSelector, which means the policy/rule is applied to Nodes, instead of Pods.
2. It's only applicable to ClusterNetworkPolicy
3. Optionally, we can enforce a validation that AppliedTo can't use NodeSelector and other selectors together to simplify the implementation.

An example of a ClusterNetworkPolicy applied to Nodes, it allows all Nodes in this cluster and clients whose IP are in 10.0.0.0/24 can access control-plane Nodes's TCP 6443 port, and drops all other accesses to these Nodes.

apiVersion: crd.antrea.io/v1alpha1
kind: ClusterNetworkPolicy
metadata:
  name: node-policy
  spec:
    priority: 5
    tier: securityops
    appliedTo:
    - nodeSelector:
        matchExpressions:
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
    ingress:
    - action: Allow
    ports:
    - port: 6443
      protocol: TCP
      from:
      - ipBlock:
          cidr: 10.0.0.0/24       # External clients
      - nodeSelector: {}          # Nodes in this cluster
  - action: Drop

Configuration Change

1. A feature gate NodeNetworkPolicy will control the enablement of the feature.
2. Considering a misconfiguration of NodeNetworkPolicies is disruptive and perhaps irreversible (for example, a deny-all rule could stop all communication between kube-apiserver and antrea components, no further policy change can be implemented), we add a collection of static rules that take priority over NodeNetworkPolicies by default, users can configure the "priviledge" rules in the configuration file if the default ones don't work for them.
3. To support applying NodeNetworkPolicy to multiple interfaces of the Node, there will be a configuration option allowing users to specify network interfaces to which NodeNetworkPoilices will be applied, e.g. [eth*, ens160]. By default, it applies to all network interfaces.

Workflow

Interfaces

In antrea-agent, we will reuse the Reconciler interface, and create a new implementation, which programs iptables and ipset to enforce the given rules.

// Reconciler is an interface that knows how to reconcile the desired state of CompletedRule.
type Reconciler interface {
	// Reconcile reconciles the desired state of the provided CompletedRule.
	Reconcile(rule *CompletedRule) error

	// BatchReconcile reconciles the desired state of the provided CompletedRules.
	BatchReconcile(rules []*CompletedRule) error

	// Forget cleans up the actual state of the specified ruleID.
	Forget(ruleID string) error
}

// NodeRuleReconciler implements Reconciler using iptables and ipset, capable of handling rule
// applied to Nodes.
type NodeRuleReconciler struct {
	iptables iptable.Interface
	ipset ipset.Interface
}

In Controller.syncRule, we will call NodeRuleReconciler instead of the existing reconciler to realize a rule if it applies to Nodes:

func (c *Controller) syncRule(key string) error {
	...
	if rule.IsAppliedToNodes() {
	    err = r.nodeRuleReconciler.Reconcile(rule)
	} else {
	    err = r.podRuleReconciler.Reconcile(rule)
	}
	...
}

Plan

• NodeNetworkPolicy targets release 1.15
  • Work breakdown:
    • Controlplane change (@Atish-iaf)
    • Dataplane support (@hongliangl)
• Unify implementation of ExternalNode on Linux (Long term)

[hongliangl]

Dataplane PRs:

• IPSet is used to store the members of source or destinations, add support of destroying ipset. Add method DestroyIPSet in pkg/agent/util/ipset #5663
• Add methods in pkg/agent/route to add or delete iptables and ipset for NodeNetworkPolicy Add methods in pkg/agent/route for NodeNetworkPolicy #5692
• Provide a more friendly way to build iptables rules Add iptables rule builder #5666
• Documents, TBD
• E2e tests, TBD

[tnqn]

@hongliangl I suppose there is a PR implementing the NodeRuleReconcoler consuming the above new util methods, is it ready for review? It's hard to judge an util is reasonable without looking at how they are consumed.

[ColonelBundy]

@tnqn Thank you for bringing some attention to this. I like your proposal and it looks like it would cover all the use cases I had from my original issue.

Well done 👍

[hongliangl]

@hongliangl I suppose there is a PR implementing the NodeRuleReconcoler consuming the above new util methods, is it ready for review? It's hard to judge an util is reasonable without looking at how they are consumed.

Please look at this PR #5658

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment, or this will be closed in 90 days

[tnqn]

Close this as completed. Related PRs: #5658 #5716