Implement multi-node cluster capabilities #6787

sharifelgamal · 2020-02-24T23:13:18Z

Fixes #6505
Begins to address #94

You can spin up a multinode cluster in 2 ways:

minikube start --nodes=2

minikube start
minikube node add

You can spin up and down individual nodes by name:

minikube node stop node2
minikube node start node2

Multi-node clusters are still experimental and won't be fully featured.

NOTE: This PR does not account for CNI, that will come in a subsequent PR. Currently, you will need to install CNI manually for multi-node to work properly.

k8s-ci-robot · 2020-02-24T23:14:22Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sharifelgamal

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [sharifelgamal]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

cmd/minikube/cmd/node_delete.go

pkg/minikube/node/node.go

pkg/minikube/config/profile.go

medyagh · 2020-02-25T00:36:51Z

cmd/minikube/cmd/node_add.go

@@ -47,16 +47,23 @@ var nodeAddCmd = &cobra.Command{
 		if nodeName == "" {
 			name = profile + strconv.Itoa(len(mc.Nodes)+1)


while we are at it, why not rename all mc to cc and make sure we call it cc everywhere in the code

codecov-io · 2020-02-25T01:34:53Z

Codecov Report

Merging #6787 into master will not change coverage by %.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #6787   +/-   ##
=======================================
  Coverage   36.98%   36.98%           
=======================================
  Files         145      145           
  Lines        9161     9161           
=======================================
  Hits         3388     3388           
  Misses       5355     5355           
  Partials      418      418

tstromberg · 2020-03-20T20:47:23Z

cmd/minikube/cmd/ssh.go

+			n, _, err = node.Retrieve(cc, nodeName)
+			if err != nil {
+				out.FailureT("Node {{.nodeName}} does not exist.", out.V{"nodeName": nodeName})
+				exit.WithError("", err)


Combine these two calls into one which generates the proper exit code:

exit.WithCodeT(exit.Unavailable, "node ... does not exist")

tstromberg · 2020-03-20T20:48:38Z

cmd/minikube/cmd/start.go

+	}
+	if numNodes > 1 {
+		if driver.BareMetal(driverName) {
+			out.T(out.Meh, "The none driver is not compatible with multi-node clusters.")


Exit rather than ignoring the user:

exit.WithCodeT(exit.Config, "....")

tstromberg · 2020-03-20T20:50:23Z

cmd/minikube/cmd/start.go

+			out.T(out.Meh, "The none driver is not compatible with multi-node clusters.")
+		} else {
+			for i := 1; i < numNodes; i++ {
+				nodeName := fmt.Sprintf("m%02d", i+1)


I noticed this formatting in two locations. Add a node.Name(int) function?

yep, added.

tstromberg · 2020-03-20T20:55:32Z

pkg/minikube/bootstrapper/kubeadm/kubeadm.go

+		return "", errors.Wrap(err, "generating bootstrap token")
+	}
+
+	/*cp, err := config.PrimaryControlPlane(&cc)


Remove debugging comment

tstromberg · 2020-03-20T21:25:52Z

/ok-to-test

minikube-pr-bot · 2020-03-20T21:42:01Z

All Times minikube: [ 65.822917 65.489519 64.712395]
All Times Minikube (PR 6787): [ 65.617288 66.311797 62.593569]

Average minikube: 65.341610
Average Minikube (PR 6787): 64.840885

Averages Time Per Log

+----------------------+-----------+--------------------+
|         LOG          | MINIKUBE  | MINIKUBE (PR 6787) |
+----------------------+-----------+--------------------+
| minikube v           |  0.228925 |           0.207446 |
| Creating kvm2        | 41.684456 |          42.867836 |
| Preparing Kubernetes |  0.754215 |          19.903093 |
| Pulling images       |           |                    |
| Launching Kubernetes | 20.794955 |                    |
| Waiting for cluster  |  0.252219 |           0.232201 |
+----------------------+-----------+--------------------+

tstromberg

I have a comment I would like to see fixed before release, but I'm OK with merging this PR to get miles on it.

Plumbing the name of the ContainerRuntime to configure into the provision library may require trickery, such as adding it to an unused field in say, SwarmOptions. I look forward to your creative solution.

tstromberg · 2020-03-20T21:36:01Z

pkg/minikube/driver/driver.go

+	return fmt.Sprintf("%s---%s", cc.Name, n.Name)
+}
+
+// ClusterNameFromMachine retrieves the cluster name embedded in the machine name


I see pain and suffering on this path. Instead of adding structure to an unstructured field & calling config.Load outside of main, we should be passing the config object around.

minikube-pr-bot · 2020-03-20T22:27:33Z

All Times minikube: [ 69.090527 65.157437 64.588555]
All Times Minikube (PR 6787): [ 65.998519 64.871788 65.604756]

Average minikube: 66.278840
Average Minikube (PR 6787): 65.491688

Averages Time Per Log

+----------------------+-----------+--------------------+
|         LOG          | MINIKUBE  | MINIKUBE (PR 6787) |
+----------------------+-----------+--------------------+
| minikube v           |  0.210650 |           0.217285 |
| Creating kvm2        | 42.059744 |          42.759002 |
| Preparing Kubernetes |  0.748763 |          20.362227 |
| Pulling images       |           |                    |
| Launching Kubernetes | 21.485618 |                    |
| Waiting for cluster  |  0.071921 |           0.229123 |
+----------------------+-----------+--------------------+

tstromberg · 2020-03-21T01:51:03Z

TestFunctional/parallel/SSHCmd failure is interesting and unexpected:

--- FAIL: TestFunctional/parallel/SSHCmd (0.35s)
functional_test.go:702: (dbg) Run:  out/minikube-linux-amd64 -p functional-20200320T163208.831234864-7487 ssh "echo hello"
functional_test.go:707: [out/minikube-linux-amd64 -p functional-20200320T163208.831234864-7487 ssh echo hello] = "hello\n", want = "hello\r\n"

sharifelgamal · 2020-03-21T02:05:22Z

I have no idea why my change to ssh would cause a carriage return to get swallowed up

sharifelgamal added 6 commits February 19, 2020 12:53

changes and things

8536fb7

let's move some start code around

ec19111

add new setup cluster file

7853387

Merge branch 'master' of github.com:kubernetes/minikube into m2

4af8291

mostly moving code around and adding UpdateNode

f22efd8

fixed more stuff

9a3ecab

sharifelgamal requested review from tstromberg and medyagh February 24, 2020 23:13

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Feb 24, 2020

k8s-ci-robot requested a review from josedonizetti February 24, 2020 23:14

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 24, 2020

medyagh reviewed Feb 24, 2020

View reviewed changes

cmd/minikube/cmd/node_delete.go Show resolved Hide resolved

medyagh reviewed Feb 24, 2020

View reviewed changes

pkg/minikube/node/node.go Outdated Show resolved Hide resolved

pass in the node object into add

be9c5f4

medyagh reviewed Feb 25, 2020

View reviewed changes

pkg/minikube/config/profile.go Outdated Show resolved Hide resolved

sharifelgamal added 2 commits February 24, 2020 16:17

fix unit tests

f0ca34b

SaveNode is simpler yeah

b2ba874

medyagh reviewed Feb 25, 2020

View reviewed changes

sharifelgamal added 3 commits February 24, 2020 16:47

fix kvm2 configurator

f5bdba6

hyperv and none drivers needed fixing too

feaa9fc

fixing lint and other random incorrect stuff

3d72152

prepareNone was in the wrong spot

39f03bc

k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Feb 25, 2020

sharifelgamal added 2 commits February 25, 2020 16:36

i think it works?

0fadf91

fix ip command

e3826a5

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 16, 2020

sharifelgamal added 9 commits March 16, 2020 15:36

fix docker driver

f99d335

fix docker driver again

84939da

fix docker status

66a6f4e

dramatically simplify start code path

a24aa5d

missing file

add1c8f

account for hyphens in profile name

efac79e

fix machine name creation

fa97a5b

clean up status output and have multinode survive cluster restarts

f9b38dc

Merge branch 'master' of github.com:kubernetes/minikube into m2

6a6b77a

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 20, 2020

Merge branch 'master' of github.com:kubernetes/minikube into m2

4c00ae0

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 20, 2020

tstromberg suggested changes Mar 20, 2020

View reviewed changes

sharifelgamal changed the title ~~WIP: Implement multi-node cluster capabilities~~ Implement multi-node cluster capabilities Mar 20, 2020

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 20, 2020

code comments

35aec77

k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Mar 20, 2020

delete admin.conf before running kubedm init

a09aa62

tstromberg approved these changes Mar 20, 2020

View reviewed changes

only apply kic networking overlay to control plane

05814cc

tstromberg approved these changes Mar 21, 2020

View reviewed changes

tstromberg merged commit 986679b into kubernetes:master Mar 21, 2020

govargo mentioned this pull request Mar 21, 2020

Cannot use local storage persistent volumes after v1.8.0 upgrade (from older v1.7.3) #7056

Closed

ngerstle mentioned this pull request Apr 10, 2020

minikube: 1.8.1 -> 1.9.2 NixOS/nixpkgs#84931

Merged

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement multi-node cluster capabilities #6787

Implement multi-node cluster capabilities #6787

sharifelgamal commented Feb 24, 2020 •

edited

Loading

k8s-ci-robot commented Feb 24, 2020

medyagh Feb 25, 2020

codecov-io commented Feb 25, 2020 •

edited

Loading

tstromberg Mar 20, 2020

sharifelgamal Mar 20, 2020

tstromberg Mar 20, 2020

sharifelgamal Mar 20, 2020

tstromberg Mar 20, 2020

sharifelgamal Mar 20, 2020

tstromberg Mar 20, 2020

sharifelgamal Mar 20, 2020

tstromberg commented Mar 20, 2020

minikube-pr-bot commented Mar 20, 2020

tstromberg left a comment

tstromberg Mar 20, 2020

minikube-pr-bot commented Mar 20, 2020

tstromberg commented Mar 21, 2020

sharifelgamal commented Mar 21, 2020

		@@ -47,16 +47,23 @@ var nodeAddCmd = &cobra.Command{
		if nodeName == "" {
		name = profile + strconv.Itoa(len(mc.Nodes)+1)

Implement multi-node cluster capabilities #6787

Implement multi-node cluster capabilities #6787

Conversation

sharifelgamal commented Feb 24, 2020 • edited Loading

k8s-ci-robot commented Feb 24, 2020

Choose a reason for hiding this comment

codecov-io commented Feb 25, 2020 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tstromberg commented Mar 20, 2020

minikube-pr-bot commented Mar 20, 2020

tstromberg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

minikube-pr-bot commented Mar 20, 2020

tstromberg commented Mar 21, 2020

sharifelgamal commented Mar 21, 2020

sharifelgamal commented Feb 24, 2020 •

edited

Loading

codecov-io commented Feb 25, 2020 •

edited

Loading