Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Add minikube support for the "generic" VM driver #9545

Closed
wants to merge 29 commits into from
Closed
Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
264072b
Add minikube support for the "generic" VM driver
afbjorklund Jul 10, 2019
63e6c66
Don't download ISO for the generic VM driver
afbjorklund Jul 14, 2019
d8712cd
Generic driver does not add user to docker group
afbjorklund Jul 14, 2019
19918ca
Try to uninstall kubernetes on delete, for generic
afbjorklund Jul 14, 2019
d53a0d3
Don't try to start/stop drivers without VMs
afbjorklund Jul 15, 2019
71e9dc1
Flip the driver logic to make code easier to read
afbjorklund Oct 28, 2020
d35d00c
Allow actually using the generic driver
afbjorklund Oct 28, 2020
55e8e33
Call DetectProvisioner for the generic driver
afbjorklund Oct 28, 2020
4197935
Show remote host info and proper progress
afbjorklund Oct 29, 2020
dc08bd8
Fix failing unit test for MachineType
afbjorklund Oct 29, 2020
9e7739e
Log the os-release also for the generic driver
afbjorklund Oct 29, 2020
3b594c3
Need to set up docker group in start - not in fix
afbjorklund Oct 29, 2020
be770af
Also parse the available, in addition to the total
afbjorklund Oct 30, 2020
e1effd3
Add some error checking on the parsed data fields
afbjorklund Oct 30, 2020
b17ecbd
Convert the return values to use uint64 instead
afbjorklund Oct 30, 2020
88df534
Supply the disk mountpoint as a parameter instead
afbjorklund Oct 30, 2020
2394e33
Merge branch 'master' into generic
afbjorklund Nov 7, 2020
251ee84
Merge branch 'master' into generic
afbjorklund Nov 14, 2020
6104fb9
fixup: out.T was renamed to out.Step
afbjorklund Nov 14, 2020
3bb75da
Revert "Supply the disk mountpoint as a parameter instead"
afbjorklund Nov 14, 2020
bb53864
Revert "Convert the return values to use uint64 instead"
afbjorklund Nov 14, 2020
612606c
Revert "Add some error checking on the parsed data fields"
afbjorklund Nov 14, 2020
08e1a65
Revert "Also parse the available, in addition to the total"
afbjorklund Nov 14, 2020
4428b3b
Revert "Show remote host info and proper progress"
afbjorklund Nov 14, 2020
70d1a31
Add helper for checking the generic driver name
afbjorklund Nov 14, 2020
1dc8769
Merge branch 'master' into generic
afbjorklund Nov 18, 2020
306d08d
Merge branch 'master' into generic
afbjorklund Nov 29, 2020
31ccaa6
Merge branch 'master' into generic
afbjorklund Dec 5, 2020
1369b08
Merge branch 'master' into generic
afbjorklund Dec 9, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion cmd/minikube/cmd/delete.go
Original file line number Diff line number Diff line change
Expand Up @@ -309,7 +309,7 @@ func deleteProfile(profile *config.Profile) error {
return DeletionError{Err: delErr, Errtype: MissingProfile}
}

if err == nil && driver.BareMetal(cc.Driver) {
if err == nil && (driver.BareMetal(cc.Driver) || cc.Driver == driver.Generic) {
if err := uninstallKubernetes(api, *cc, cc.Nodes[0], viper.GetString(cmdcfg.Bootstrapper)); err != nil {
deletionError, ok := err.(DeletionError)
if ok {
Expand Down
2 changes: 1 addition & 1 deletion cmd/minikube/cmd/start.go
Original file line number Diff line number Diff line change
Expand Up @@ -299,7 +299,7 @@ func provisionWithDriver(cmd *cobra.Command, ds registry.DriverState, existing *
os.Exit(0)
}

if driver.IsVM(driverName) {
if driver.IsVM(driverName) && driverName != driver.Generic {
url, err := download.ISO(viper.GetStringSlice(isoURL), cmd.Flags().Changed(isoURL))
if err != nil {
return node.Starter{}, errors.Wrap(err, "Failed to cache ISO")
Expand Down
15 changes: 15 additions & 0 deletions cmd/minikube/cmd/start_flags.go
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ import (
"time"

"github.com/blang/semver"
"github.com/docker/machine/libmachine/drivers"
"github.com/pkg/errors"
"github.com/spf13/cobra"
"github.com/spf13/viper"
Expand Down Expand Up @@ -108,6 +109,10 @@ const (
kicBaseImage = "base-image"
startOutput = "output"
ports = "ports"
genericIPAddress = "generic-ip-address"
genericSSHUser = "generic-ssh-user"
genericSSHKey = "generic-ssh-key"
genericSSHPort = "generic-ssh-port"
)

// initMinikubeFlags includes commandline flags for minikube.
Expand Down Expand Up @@ -212,6 +217,12 @@ func initNetworkingFlags() {
startCmd.Flags().String(serviceCIDR, constants.DefaultServiceCIDR, "The CIDR to be used for service cluster IPs.")
startCmd.Flags().StringArrayVar(&config.DockerEnv, "docker-env", nil, "Environment variables to pass to the Docker daemon. (format: key=value)")
startCmd.Flags().StringArrayVar(&config.DockerOpt, "docker-opt", nil, "Specify arbitrary flags to pass to the Docker daemon. (format: key=value)")

// generic
startCmd.Flags().String(genericIPAddress, "", "IP address (generic)")

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should validate that all of these flags are set and exit with a helpful error message if the user has forgotten anything

Copy link
Collaborator Author

@afbjorklund afbjorklund Oct 28, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did that, but it was lost during the rebase. It should at least give an error if forgetting to provide an IP address ?

The other parameters are optional, by default it will connect as root on port 22 and assume keys are in ssh-agent

startCmd.Flags().String(genericSSHUser, drivers.DefaultSSHUser, "SSH user (generic)")
startCmd.Flags().String(genericSSHKey, "", "SSH key (generic)")
startCmd.Flags().Int(genericSSHPort, drivers.DefaultSSHPort, "SSH port (generic)")
}

// ClusterFlagValue returns the current cluster name based on flags
Expand Down Expand Up @@ -321,6 +332,10 @@ func generateClusterConfig(cmd *cobra.Command, existing *config.ClusterConfig, k
NatNicType: viper.GetString(natNicType),
StartHostTimeout: viper.GetDuration(waitTimeout),
ExposedPorts: viper.GetStringSlice(ports),
GenericIPAddress: viper.GetString(genericIPAddress),
GenericSSHUser: viper.GetString(genericSSHUser),
GenericSSHKey: viper.GetString(genericSSHKey),
GenericSSHPort: viper.GetInt(genericSSHPort),
KubernetesConfig: config.KubernetesConfig{
KubernetesVersion: k8sVersion,
ClusterName: ClusterFlagValue(),
Expand Down
6 changes: 6 additions & 0 deletions pkg/minikube/cluster/ip.go
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,12 @@ func HostIP(host *host.Host, clusterName string) (net.IP, error) {
return oci.RoutableHostIPFromInside(oci.Docker, clusterName, host.Name)
case driver.Podman:
return oci.RoutableHostIPFromInside(oci.Podman, clusterName, host.Name)
case driver.Generic:
ip, err := host.Driver.GetIP()
if err != nil {
return []byte{}, errors.Wrap(err, "Error getting VM/Host IP address")
}
return net.ParseIP(ip), nil
case driver.KVM2:
return net.ParseIP("192.168.39.1"), nil
case driver.HyperV:
Expand Down
4 changes: 4 additions & 0 deletions pkg/minikube/config/types.go
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,10 @@ type ClusterConfig struct {
HostDNSResolver bool // Only used by virtualbox
HostOnlyNicType string // Only used by virtualbox
NatNicType string // Only used by virtualbox
GenericIPAddress string // Only used by generic
GenericSSHUser string // Only used by generic
GenericSSHKey string // Only used by generic
GenericSSHPort int // Only used by generic
KubernetesConfig KubernetesConfig
Nodes []Node
Addons map[string]bool
Expand Down
2 changes: 2 additions & 0 deletions pkg/minikube/driver/driver.go
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,8 @@ const (
Mock = "mock"
// None driver
None = "none"
// Generic driver
Generic = "generic"
// KVM2 driver
KVM2 = "kvm2"
// VirtualBox driver
Expand Down
48 changes: 31 additions & 17 deletions pkg/minikube/machine/fix.go
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,12 @@ func fixHost(api libmachine.API, cc *config.ClusterConfig, n *config.Node) (*hos
return h, nil
}

if h.Driver.DriverName() == driver.Generic {
if _, err := h.RunSSHCommand(fmt.Sprintf("sudo usermod -aG docker %s", h.Driver.GetSSHUsername())); err != nil {
return h, errors.Wrap(err, "usermod")
}
}

return h, ensureSyncedGuestClock(h, driverName)
}

Expand Down Expand Up @@ -129,26 +135,34 @@ func recreateIfNeeded(api libmachine.API, cc *config.ClusterConfig, n *config.No
}
}

if serr != constants.ErrMachineMissing {
klog.Warningf("unexpected machine state, will restart: %v", serr)
}
if h.Driver.DriverName() != driver.Generic {
if serr != constants.ErrMachineMissing {
klog.Warningf("unexpected machine state, will restart: %v", serr)
}

if s == state.Running {
if !recreated {
out.T(style.Running, `Updating the running {{.driver_name}} "{{.cluster}}" {{.machine_type}} ...`, out.V{"driver_name": cc.Driver, "cluster": machineName, "machine_type": machineType})
if s == state.Running {
if !recreated {
out.T(style.Running, `Updating the running {{.driver_name}} "{{.cluster}}" {{.machine_type}} ...`, out.V{"driver_name": cc.Driver, "cluster": machineName, "machine_type": machineType})
}
return h, nil
}
return h, nil
}

if !recreated {
out.T(style.Restarting, `Restarting existing {{.driver_name}} {{.machine_type}} for "{{.cluster}}" ...`, out.V{"driver_name": cc.Driver, "cluster": machineName, "machine_type": machineType})
}
if err := h.Driver.Start(); err != nil {
MaybeDisplayAdvice(err, h.DriverName)
return h, errors.Wrap(err, "driver start")
}
if err := saveHost(api, h, cc, n); err != nil {
return h, err
if !recreated {
out.T(style.Restarting, `Restarting existing {{.driver_name}} {{.machine_type}} for "{{.cluster}}" ...`, out.V{"driver_name": cc.Driver, "cluster": machineName, "machine_type": machineType})
}
if err := h.Driver.Start(); err != nil {
MaybeDisplayAdvice(err, h.DriverName)
return h, errors.Wrap(err, "driver start")
}
if err := saveHost(api, h, cc, n); err != nil {
return h, err
}
} else {
afbjorklund marked this conversation as resolved.
Show resolved Hide resolved
if s == state.Running {
out.T(style.Running, `Using the {{.driver_name}} "{{.cluster}}" {{.machine_type}} ...`, out.V{"driver_name": cc.Driver, "cluster": cc.Name, "machine_type": machineType})
} else {
return h, errors.Errorf("not running")
}
}

return h, nil
Expand Down
16 changes: 9 additions & 7 deletions pkg/minikube/machine/stop.go
Original file line number Diff line number Diff line change
Expand Up @@ -55,14 +55,16 @@ func stop(h *host.Host) error {
}
}

if err := h.Stop(); err != nil {
klog.Infof("stop err: %v", err)
st, ok := err.(mcnerror.ErrHostAlreadyInState)
if ok && st.State == state.Stopped {
klog.Infof("host is already stopped")
return nil
if h.DriverName != driver.Generic {
afbjorklund marked this conversation as resolved.
Show resolved Hide resolved
if err := h.Stop(); err != nil {
klog.Infof("stop err: %v", err)
st, ok := err.(mcnerror.ErrHostAlreadyInState)
if ok && st.State == state.Stopped {
klog.Infof("host is already stopped")
return nil
}
return &retry.RetriableError{Err: errors.Wrap(err, "stop")}
}
return &retry.RetriableError{Err: errors.Wrap(err, "stop")}
}
klog.Infof("duration metric: stop complete within %s", time.Since(start))
return nil
Expand Down
62 changes: 62 additions & 0 deletions pkg/minikube/registry/drvs/generic/generic.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
/*
Copyright 2019 The Kubernetes Authors All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package generic

import (
"fmt"

"github.com/docker/machine/drivers/generic"
"github.com/docker/machine/libmachine/drivers"
"github.com/pkg/errors"

"k8s.io/minikube/pkg/minikube/config"
"k8s.io/minikube/pkg/minikube/driver"
"k8s.io/minikube/pkg/minikube/localpath"
"k8s.io/minikube/pkg/minikube/registry"
)

func init() {
err := registry.Register(registry.DriverDef{
Name: driver.Generic,
Config: configure,
Status: status,
Priority: registry.Fallback,
Init: func() drivers.Driver { return generic.NewDriver("", "") },
})
if err != nil {
panic(fmt.Sprintf("unable to register: %v", err))
}
}

func configure(cc config.ClusterConfig, n config.Node) (interface{}, error) {
d := generic.NewDriver(driver.MachineName(cc, n), localpath.MiniPath())

if cc.GenericIPAddress == "" {
return nil, errors.Errorf("please provide an IP address")
}

d.(*generic.Driver).IPAddress = cc.GenericIPAddress
d.(*generic.Driver).SSHUser = cc.GenericSSHUser
d.(*generic.Driver).SSHKey = cc.GenericSSHKey
d.(*generic.Driver).SSHPort = cc.GenericSSHPort

return d, nil
}

func status() registry.State {
return registry.State{Installed: true, Healthy: true}
}
1 change: 1 addition & 0 deletions pkg/minikube/registry/drvs/init.go
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ package drvs
import (
// Register all of the drvs we know of
_ "k8s.io/minikube/pkg/minikube/registry/drvs/docker"
_ "k8s.io/minikube/pkg/minikube/registry/drvs/generic"
_ "k8s.io/minikube/pkg/minikube/registry/drvs/hyperkit"
_ "k8s.io/minikube/pkg/minikube/registry/drvs/hyperv"
_ "k8s.io/minikube/pkg/minikube/registry/drvs/kvm2"
Expand Down
4 changes: 4 additions & 0 deletions site/content/en/docs/commands/start.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,10 @@ minikube start [flags]
--feature-gates string A set of key=value pairs that describe feature gates for alpha/experimental features.
--force Force minikube to perform possibly dangerous operations
--force-systemd If set, force the container runtime to use sytemd as cgroup manager. Currently available for docker and crio. Defaults to false.
--generic-ip-address string IP address (generic)
--generic-ssh-key string SSH key (generic)
--generic-ssh-port int SSH port (generic) (default 22)
--generic-ssh-user string SSH user (generic) (default "root")
-h, --help help for start
--host-dns-resolver Enable host resolver for NAT DNS requests (virtualbox driver only) (default true)
--host-only-cidr string The CIDR to be used for the minikube VM (virtualbox driver only) (default "192.168.99.1/24")
Expand Down