docker fails if --docker-opt storage-driver=overlay2 is set #7070

fredsted · 2020-03-17T09:18:13Z

Started having this problem after upgrading to 1.8.2.

It disappeared after downgrading to back to 1.7.x. Could not find anything recent regarding this anywhere else.

I've tried deleting the ~/.minikube folder and reinstalling minikube via homebrew. Didn't help.

Example of output (see link to full logs below)

W0317 10:02:52.019784    1773 exit.go:101] Unable to start VM. Please investigate and run 'minikube delete' if possible: creating host: create: Error creating machine: Error running provisioning: ssh command error:
command : sudo systemctl -f restart docker
err     : Process exited with status 1
output  : Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xe" for details.

The exact command to reproduce the issue:

minikube start --v 10 --logtostderr \
  --disk-size="20G" \
  --cpus="4" \
  --docker-opt storage-driver=overlay2 \
  --extra-config=apiserver.authorization-mode="RBAC" \
  --kubernetes-version v1.15.5 \
  --memory="5120" \
  --network-plugin=cni \
  --enable-default-cni \
  --vm-driver="hyperkit"

The full output of the command that failed:

See here: https://sf.gl/files/minikube-out.txt

The operating system version:
macOS 10.15.3

The text was updated successfully, but these errors were encountered:

tstromberg · 2020-03-18T18:25:59Z

Interesting! Can you share the output of:

minikube logs
minikube ssh "systemctl status docker.service"

Thanks!

Also, can you check if removing this flag fixes your issue:

--docker-opt storage-driver=overlay2

fredsted · 2020-03-19T07:53:19Z

Thanks for the quick response.

Removing the --docker-opt storage-driver=overlay2 flag solved the issue!

Here's the minikube logs output: https://sf.gl/files/minikube-logs-output.txt – unfortunately couldn't get more since the VM isn't starting.

Looks like since the storage driver configuration property is duplicated, it errors out?

tstromberg · 2020-03-19T20:23:42Z

@fredsted - Indeed, it seems this is the case:

Mar 19 07:38:09 minikube dockerd[2034]: unable to configure the Docker daemon with file /etc/docker/daemon.json: the following directives are specified both as a flag and in the configuration file: storage-driver: (from flag: overlay2, from file: overlay2)

I believe that this happened in v1.8.x because minikube now needs overlay2 for the image preload optimizations.

It looks like we could do something smarter about this.

pkuma001 · 2020-05-17T21:18:43Z

Hi,

I also have a same problem while executing minikube start.

I have virtualbox in my Windows 10 laptop. I am trying to create a K8s cluster using minikube start and having the below issue,

PS C:\Program Files\Kubernetes\Minikube> minikube start --iso-url=https://storage.googleapis.com/minikube/iso/minikube-v1.7.0.iso --no-vtx-check --driver=virtualbox

minikube v1.9.2 on Microsoft Windows 10 Enterprise 10.0.18362 Build 18362
Using the virtualbox driver based on user configuration
Downloading VM boot image ...

minikube-v1.7.0.iso: 166.68 MiB / 166.68 MiB 100.00% 203.51 KiB p/s 13m5
Starting control plane node m01 in cluster minikube
Creating virtualbox VM (CPUs=2, Memory=6000MB, Disk=20000MB) ...
! StartHost failed, but will try again: creating host: create: provisioning: ssh command error:
command : sudo diff -u /lib/systemd/system/docker.service /lib/systemd/system/docker.service.new || { sudo mv /lib/systemd/system/docker.service.new /lib/systemd/system/docker.service; sudo systemctl -f daemon-reload && sudo sudo systemctl -f restart docker; }
err : Process exited with status 1
output : diff: can't stat '/lib/systemd/system/docker.service': No such file or directory
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xe" for details.

Deleting "minikube" in virtualbox ...
Creating virtualbox VM (CPUs=2, Memory=6000MB, Disk=20000MB) ...
[DOCKER_RESTART_FAILED] Failed to start virtualbox VM. "minikube start" may fix it. creating host: create: provisioning: ssh command error:
command : sudo diff -u /lib/systemd/system/docker.service /lib/systemd/system/docker.service.new || { sudo mv /lib/systemd/system/docker.service.new /lib/systemd/system/docker.service; sudo systemctl -f daemon-reload && sudo sudo systemctl -f restart docker; }
err : Process exited with status 1
output : diff: can't stat '/lib/systemd/system/docker.service': No such file or directory
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xe" for details.

Suggestion: Remove the incompatible --docker-opt flag if one was provided
Related issue: #7070

Other information,

This was working fine for me two weeks before.
I tried to set --docker-opt to storage-driver=overlay still I have the problem. I dont know howelse I can remove --docker-opt flag?
Tried to install olderversion of minikube.iso using --iso-url=https://storage.googleapis.com/minikube/iso/minikube-v1.7.0.iso and still I have a problem.
Tried with version 1.10.0 and still issue exists.

Please do let me know if you have a fix or workaround.

medyagh · 2020-05-20T21:26:54Z

@fredsted
@pkuma001
Why do have to pass -docker-opt to storage-driver=overlay ? the docker inside minikube already is using overlay2 driver, you dont need to specify it.

I also noticed you are using an older minikube, have u tried with our latest version ?
also i noticed u are using virtualbox driver, have u tried docker driver?

pkuma001 · 2020-05-21T04:34:46Z

@medyagh as i mentioned in my comment the latest version of minikube I tried is 1.10.0. And still the problem exists in that version.

And I did not try the docker driver. I tried Hyper-v and it works fine.

afbjorklund · 2020-05-21T10:39:28Z

@medyagh @pkuma001 : please open a new issue about virtualbox/hyper-v issues, thanks.

We should probably not issue that "suggestion", when there is no docker-opt being provided.
Suggestion: Remove the incompatible --docker-opt flag if one was provided

Should just "merge" that option with the existing default ones, to notice that they are the same.

Minikube doesn't "need" overlay2 anyway, so it is more or less redundant

The preload should be smart enough to check the driver, and fall back #7626 #7933

jonniespratley · 2020-06-05T17:11:39Z

Any tips on this issue.

+ echo 'Starting minikube...'
Starting minikube...
+ minikube stop
✋  Stopping "minikube" in virtualbox ...
🛑  Node "minikube" stopped.
+ minikube delete
🔥  Deleting "minikube" in virtualbox ...
💀  Removed all traces of the "minikube" cluster.
+ minikube start --vm-driver virtualbox --docker-env http_proxy=http://grc-americas-sanra-pitc-wkcz.proxy.corporate.gtm.ge.com:80 --docker-env https_proxy=http://grc-americas-sanra-pitc-wkcz.proxy.corporate.gtm.ge.com:80 --docker-env 'no_proxy=localhost,*.minikube,10.96.0.0/12,192.168.99.0/24,192.168.39.0/24,127.0.0.1,169.254/16"
😄  minikube v1.11.0 on Darwin 10.13.6
✨  Using the virtualbox driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🔥  Creating virtualbox VM (CPUs=4, Memory=8096MB, Disk=20000MB) ...
🔥  Deleting "minikube" in virtualbox ...
🤦  StartHost failed, but will try again: creating host: create: provisioning: ssh command error:
command : sudo diff -u /lib/systemd/system/docker.service /lib/systemd/system/docker.service.new || { sudo mv /lib/systemd/system/docker.service.new /lib/systemd/system/docker.service; sudo systemctl -f daemon-reload && sudo systemctl -f enable docker && sudo systemctl -f restart docker; }
err     : Process exited with status 1
output  : diff: can't stat '/lib/systemd/system/docker.service': No such file or directory
Failed to enable unit: Unit file /usr/lib/systemd/system/docker.service is masked.

🔥  Creating virtualbox VM (CPUs=4, Memory=8096MB, Disk=20000MB) ...
😿  Failed to start virtualbox VM. "minikube start" may fix it: creating host: create: provisioning: ssh command error:
command : sudo diff -u /lib/systemd/system/docker.service /lib/systemd/system/docker.service.new || { sudo mv /lib/systemd/system/docker.service.new /lib/systemd/system/docker.service; sudo systemctl -f daemon-reload && sudo systemctl -f enable docker && sudo systemctl -f restart docker; }
err     : Process exited with status 1
output  : diff: can't stat '/lib/systemd/system/docker.service': No such file or directory
Failed to enable unit: Unit file /usr/lib/systemd/system/docker.service is masked.


❌  [DOCKER_RESTART_FAILED] error provisioning host Failed to start host: creating host: create: provisioning: ssh command error:
command : sudo diff -u /lib/systemd/system/docker.service /lib/systemd/system/docker.service.new || { sudo mv /lib/systemd/system/docker.service.new /lib/systemd/system/docker.service; sudo systemctl -f daemon-reload && sudo systemctl -f enable docker && sudo systemctl -f restart docker; }
err     : Process exited with status 1
output  : diff: can't stat '/lib/systemd/system/docker.service': No such file or directory
Failed to enable unit: Unit file /usr/lib/systemd/system/docker.service is masked.

💡  Suggestion: Remove the incompatible --docker-opt flag if one was provided
⁉️   Related issue: https://github.com/kubernetes/minikube/issues/7070

keyhan · 2020-07-15T19:31:57Z

Any suggestions for the newbies how to get past this?

medyagh · 2020-07-15T19:44:24Z

@keyhan are you using the docker driver. is there a reason you set --docker-opt storage-driver=overlay2 option ?

keyhan · 2020-07-15T19:57:42Z

Hi, I am using ubuntu installation of docker, have not touched it. just done "apt install docker-io". Right now I am upgrading to the latest release to see if it solves the problem.

…

On Wed, Jul 15, 2020 at 9:44 PM Medya Ghazizadeh ***@***.***> wrote: @keyhan <https://github.com/keyhan> are you using the docker driver. is there a reason you set --docker-opt storage-driver=overlay2 option ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7070 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIHGM5FXG2DEQ5B2JOP5J3R3YBKRANCNFSM4LNGMV6A> .

-- ----------------------- Best Regards Keyhan Hadjari

MisLink · 2020-07-29T16:34:53Z

Error message bring me here... Although I don't think I have the same issue.

I start minikube by

minikube start --driver=vmware --registry-mirror=https://bdgkyxdw.mirror.aliyuncs.com --image-mirror-country=cn --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers --iso-url=https://kubernetes.oss-cn-hangzhou.aliyuncs.com/minikube/iso/minikube-v1.12.0.iso --kubernetes-version=stable

Output is

😄  minikube v1.12.1 on Darwin 10.15.5
✨  Using the vmware driver based on user configuration
✅  Using image repository registry.cn-hangzhou.aliyuncs.com/google_containers
👍  Starting control plane node minikube in cluster minikube
🔥  Creating vmware VM (CPUs=2, Memory=6000MB, Disk=20000MB) ...
🔥  Deleting "minikube" in vmware ...
🤦  StartHost failed, but will try again: creating host: create: Error creating machine: Error running provisioning: ssh command error:
command : sudo diff -u /lib/systemd/system/docker.service /lib/systemd/system/docker.service.new || { sudo mv /lib/systemd/system/docker.service.new /lib/systemd/system/docker.service; sudo systemctl -f daemon-reload && sudo systemctl -f enable docker && sudo systemctl -f restart docker; }
err     : Process exited with status 1
output  : diff: can't stat '/lib/systemd/system/docker.service': No such file or directory
Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /usr/lib/systemd/system/docker.service.
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xe" for details.

🔥  Creating vmware VM (CPUs=2, Memory=6000MB, Disk=20000MB) ...
😿  Failed to start vmware VM. "minikube start" may fix it: creating host: create: Error creating machine: Error running provisioning: ssh command error:
command : sudo diff -u /lib/systemd/system/docker.service /lib/systemd/system/docker.service.new || { sudo mv /lib/systemd/system/docker.service.new /lib/systemd/system/docker.service; sudo systemctl -f daemon-reload && sudo systemctl -f enable docker && sudo systemctl -f restart docker; }
err     : Process exited with status 1
output  : diff: can't stat '/lib/systemd/system/docker.service': No such file or directory
Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /usr/lib/systemd/system/docker.service.
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xe" for details.


❌  [DOCKER_RESTART_FAILED] error provisioning host Failed to start host: creating host: create: Error creating machine: Error running provisioning: ssh command error:
command : sudo diff -u /lib/systemd/system/docker.service /lib/systemd/system/docker.service.new || { sudo mv /lib/systemd/system/docker.service.new /lib/systemd/system/docker.service; sudo systemctl -f daemon-reload && sudo systemctl -f enable docker && sudo systemctl -f restart docker; }
err     : Process exited with status 1
output  : diff: can't stat '/lib/systemd/system/docker.service': No such file or directory
Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /usr/lib/systemd/system/docker.service.
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xe" for details.

💡  Suggestion: Remove the incompatible --docker-opt flag if one was provided
⁉️   Related issue: https://github.com/kubernetes/minikube/issues/7070

I don't use docker driver and docker-opt option. I tried to run minikube start again but not luck.

Finally I try to delete the .minikube/ and .kube/ folder, downgrade the iso to v1.11.0, then minikube works fine.

fejta-bot · 2020-10-27T17:34:24Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot · 2020-11-26T18:17:18Z

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

fejta-bot · 2020-12-26T19:01:59Z

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

k8s-ci-robot · 2020-12-26T19:02:11Z

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

tstromberg added co/runtime/docker Issues specific to a docker runtime triage/needs-information Indicates an issue needs more information in order to work on it. kind/support Categorizes issue or PR as a support question. labels Mar 18, 2020

tstromberg added the needs-solution-message Issues where where offering a solution for an error would be helpful label Mar 19, 2020

tstromberg changed the title ~~"ssh command error: command : sudo systemctl -f restart docker" while starting~~ sudo systemctl -f restart docke fails if --docker-opt storage-driver=overlay2 is specified Mar 19, 2020

tstromberg changed the title ~~sudo systemctl -f restart docke fails if --docker-opt storage-driver=overlay2 is specified~~ docker fails if --docker-opt storage-driver=overlay2 is set Mar 19, 2020

tstromberg removed the needs-solution-message Issues where where offering a solution for an error would be helpful label Apr 2, 2020

tstromberg mentioned this issue Apr 2, 2020

Update err map for v1.9.x #7390

Merged

elegos mentioned this issue May 4, 2020

podman: container engine docker: StartHost failed #7992

Closed

medyagh added the triage/needs-information Indicates an issue needs more information in order to work on it. label May 20, 2020

anencore94 mentioned this issue Jul 21, 2020

bad --insecure-registry values cause: Job for docker.service failed because the control process exited with error code. #8790

Closed

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 27, 2020

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Nov 26, 2020

k8s-ci-robot closed this as completed Dec 26, 2020

rafacouto mentioned this issue Sep 23, 2021

Exiting due to K8S_KUBELET_NOT_RUNNING #11932

Closed

samarjit mentioned this issue Jul 18, 2022

Minikube virtualbox vm has docker.service masked #14605

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docker fails if --docker-opt storage-driver=overlay2 is set #7070

docker fails if --docker-opt storage-driver=overlay2 is set #7070

fredsted commented Mar 17, 2020 •

edited

Loading

tstromberg commented Mar 18, 2020

fredsted commented Mar 19, 2020

tstromberg commented Mar 19, 2020

pkuma001 commented May 17, 2020 •

edited

Loading

medyagh commented May 20, 2020 •

edited

Loading

pkuma001 commented May 21, 2020

afbjorklund commented May 21, 2020

jonniespratley commented Jun 5, 2020

keyhan commented Jul 15, 2020

medyagh commented Jul 15, 2020

keyhan commented Jul 15, 2020 via email

MisLink commented Jul 29, 2020

fejta-bot commented Oct 27, 2020

fejta-bot commented Nov 26, 2020

fejta-bot commented Dec 26, 2020

k8s-ci-robot commented Dec 26, 2020

docker fails if --docker-opt storage-driver=overlay2 is set #7070

docker fails if --docker-opt storage-driver=overlay2 is set #7070

Comments

fredsted commented Mar 17, 2020 • edited Loading

tstromberg commented Mar 18, 2020

fredsted commented Mar 19, 2020

tstromberg commented Mar 19, 2020

pkuma001 commented May 17, 2020 • edited Loading

medyagh commented May 20, 2020 • edited Loading

pkuma001 commented May 21, 2020

afbjorklund commented May 21, 2020

jonniespratley commented Jun 5, 2020

keyhan commented Jul 15, 2020

medyagh commented Jul 15, 2020

keyhan commented Jul 15, 2020 via email

MisLink commented Jul 29, 2020

fejta-bot commented Oct 27, 2020

fejta-bot commented Nov 26, 2020

fejta-bot commented Dec 26, 2020

k8s-ci-robot commented Dec 26, 2020

fredsted commented Mar 17, 2020 •

edited

Loading

pkuma001 commented May 17, 2020 •

edited

Loading

medyagh commented May 20, 2020 •

edited

Loading