datapath/linux: New scalable routing layer via Linux datapath impleme…

…ntation

Scale testing beyond 1k nodes have lead to the conclusion that the interactions
with the Linux routing tables have been inefficient. Several small factors are
contributing to the overhead with the main reason behind naive routing updates
for the entire list of nodes on certain node events.

pkg/datapath/linux implements the Linux specific datapath portion for node
events handling according to the node configuration. This includes
determination on what type of routes to push down based on enablement of
address families, encapsulation, direct-routing and use of single cluster
prefixes.

The datapath layer is responsible to optimize detection of changes in node
information and only perform routing layer changes when the definition has
changed. Changes include:

* A node changing its IP address
* A node starting or stopping to announce an allocation CIDR
* A node changing its allocation CIDR

The existing AutoIPv6NodeRoutes mode is made more generic and automated. It now
applies to IPv6 and IPv4 and no longer requires to specify any device
information. When EnableAutoDirectRouting is enabled, the datapath layer will
detect whether a direct routing connection is possible between two nodes and
install a direct node route automatically. If the direct routing is not
feasible, likely because the remote node does not share the same L2 segment,
fallback to encapsulation is performed.

Cost of node update event when node information is not changing:

```
BenchmarkNoChangeNodeUpdate                       5000000            628 ns/op
BenchmarkNoChangeNodeUpdateDirectRouteAll         5000000            699 ns/op
BenchmarkNoChangeNodeUpdateEncapAll                500000           3637 ns/op
```

Cost of node update event when node CIDR is changing. This is the cost paid
when a node is added, the node IP changes or the node changes its CIDR.

```
BenchmarkNodeUpdateAll                            5000000            639 ns/op
BenchmarkNodeUpdateIPv4                           5000000            618 ns/op
BenchmarkNodeUpdateIPv6                           5000000            662 ns/op
BenchmarkNodeUpdateDirectRouteAll                    1000        1131640 ns/op
BenchmarkNodeUpdateDirectRouteIPv4                   5000         557249 ns/op
BenchmarkNodeUpdateDirectRouteIPv6                   5000         661931 ns/op
BenchmarkNodeUpdateEncapAll                          1000        1915020 ns/op
BenchmarkNodeUpdateEncapIPv4                         1000        1007195 ns/op
BenchmarkNodeUpdateEncapIPv6                         1000        1374214 ns/op
BenchmarkNodeUpdateEncapSingleClusterRouteAll      200000          12388 ns/op
BenchmarkNodeUpdateEncapSingleClusterRouteIPv4     200000           7680 ns/op
BenchmarkNodeUpdateEncapSingleClusterRouteIPv6     200000           6625 ns/op
```

Cost of validation to determine whether a node is correctly implemented in the
datapath. This is the cost paid per node via the controller to periodically
check the datapath implementation.

```
BenchmarkNodeValidateImplementationAll                        2000         813551 ns/op
BenchmarkNodeValidateImplementationIPv4                       2000         839429 ns/op
BenchmarkNodeValidateImplementationIPv6                       2000         980503 ns/op
BenchmarkNodeValidateImplementationDirectRouteAll             2000         861470 ns/op
BenchmarkNodeValidateImplementationDirectRouteIPv4            2000         552912 ns/op
BenchmarkNodeValidateImplementationDirectRouteIPv6            5000         487668 ns/op
BenchmarkNodeValidateImplementationEncapAll                   1000        1286646 ns/op
BenchmarkNodeValidateImplementationEncapIPv4                  5000         722960 ns/op
BenchmarkNodeValidateImplementationEncapIPv6                  2000         791391 ns/op
BenchmarkNodeValidateImplementationEncapSingleClusterAll      5000         365746 ns/op
BenchmarkNodeValidateImplementationEncapSingleClusterIPv4     5000         377625 ns/op
BenchmarkNodeValidateImplementationEncapSingleClusterIPv6     5000         427404 ns/op
```

Signed-off-by: Thomas Graf <[email protected]>

pkg/datapath/linux/datapath.go

@@ -0,0 +1,55 @@
+// Copyright 2019 Authors of Cilium
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package linux
+
+import (
+	"github.com/cilium/cilium/pkg/datapath"
+)
+
+// DatapathConfiguration is the static configuration of the datapath. The
+// configuration cannot change throughout the lifetime of a datapath object.
+type DatapathConfiguration struct {
+	// HostDevice is the name of the device to be used to access the host.
+	HostDevice string
+}
+
+type linuxDatapath struct {
+	node           datapath.NodeHandler
+	nodeAddressing datapath.NodeAddressing
+	config         DatapathConfiguration
+}
+
+// NewDatapath creates a new Linux datapath
+func NewDatapath(config DatapathConfiguration) datapath.Datapath {
+	dp := &linuxDatapath{
+		nodeAddressing: NewNodeAddressing(),
+		config:         config,
+	}
+
+	dp.node = NewNodeHandler(config, dp.nodeAddressing)
+
+	return dp
+}
+
+// Node returns the handler for node events
+func (l *linuxDatapath) Node() datapath.NodeHandler {
+	return l.node
+}
+
+// LocalNodeAddressing returns the node addressing implementation of the local
+// node
+func (l *linuxDatapath) LocalNodeAddressing() datapath.NodeAddressing {
+	return l.nodeAddressing
+}

pkg/datapath/linux/datapath_test.go

@@ -0,0 +1,39 @@
+// Copyright 2019 Authors of Cilium
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// +build !privileged_tests
+
+package linux
+
+import (
+	"testing"
+
+	"gopkg.in/check.v1"
+)
+
+func Test(t *testing.T) {
+	check.TestingT(t)
+}
+
+type linuxTestSuite struct{}
+
+var _ = check.Suite(&linuxTestSuite{})
+
+func (s *linuxTestSuite) TestNewDatapath(c *check.C) {
+	dp := NewDatapath(DatapathConfiguration{})
+	c.Assert(dp, check.Not(check.IsNil))
+
+	c.Assert(dp.Node(), check.Not(check.IsNil))
+	c.Assert(dp.LocalNodeAddressing(), check.Not(check.IsNil))
+}

pkg/datapath/linux/doc.go

@@ -0,0 +1,16 @@
+// Copyright 2019 Authors of Cilium
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package linux implements the Linux specific datapath implementation
+package linux

pkg/datapath/linux/logfields.go

@@ -0,0 +1,22 @@
+// Copyright 2019 Authors of Cilium
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package linux
+
+import (
+	"github.com/cilium/cilium/pkg/logging"
+	"github.com/cilium/cilium/pkg/logging/logfields"
+)
+
+var log = logging.DefaultLogger.WithField(logfields.LogSubsys, "linux-datapath")