Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docker v1.12: <index (index .NetworkSettings.Ports "22/tcp") 0>: error calling index: index of untyped nil -> container exited unexpectedly #8814

Closed
etanshaul opened this issue Jul 23, 2020 · 12 comments
Closed

docker v1.12: <index (index .NetworkSettings.Ports "22/tcp") 0>: error calling index: index of untyped nil -> container exited unexpectedly #8814

etanshaul opened this issue Jul 23, 2020 · 12 comments
Assignees
@medyagh
Labels
co/docker-driver Issues related to kubernetes in container july-chill kind/bug Categorizes issue or PR as related to a bug. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.

Comments

@etanshaul
Copy link

Version 1.12.0

See this comment for the full steps taken: https://github.com/GoogleCloudPlatform/cloud-code-intellij-internal/issues/2207#issuecomment-663070544

This may have since been fixed?

/Users/balintp/Library/Application Support/google-cloud-tools-java/managed-cloud-sdk/LATEST/google-cloud-sdk/bin/minikube start --profile cloud-code-minikube --keep-context true --wait false --vm-driver docker --interactive false --delete-on-failure
* [cloud-code-minikube] minikube v1.12.0 on Darwin 10.15.5
  - MINIKUBE_WANTUPDATENOTIFICATION=false
* Using the docker driver based on existing profile
* Starting control plane node cloud-code-minikube in cluster cloud-code-minikube
* Downloading Kubernetes v1.18.3 preload ...
    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 2.64 MiB /    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 9.11 MiB /    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 15.62 MiB     > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 21.86 MiB     > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 28.58 MiB     > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 34.14 MiB     > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 40.03 MiB     > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 46.08 MiB     > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 51.48 MiB     > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 56.00 MiB     > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 60.69 MiB     > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 67.44 MiB     > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 74.19 MiB     > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 79.30 MiB     > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 83.64 MiB     > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 88.00 MiB     > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 93.45 MiB     > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 99.97 MiB     > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 106.69 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 113.45 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 120.02 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 126.61 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 133.23 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 139.78 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 146.38 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 152.83 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 159.38 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 166.08 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 172.80 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 179.22 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 185.80 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 192.52 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 199.11 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 205.77 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 212.36 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 219.25 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 223.86 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 229.92 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 236.69 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 243.36 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 249.31 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 255.98 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 262.61 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 269.31 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 275.98 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 282.31 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 288.98 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 295.64 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 301.12 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 307.22 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 308.59 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 309.39 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 316.09 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 322.81 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 329.56 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 336.23 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 342.95 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 349.70 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 356.47 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 363.19 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 368.92 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 373.45 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 379.92 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 386.61 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 393.31 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 398.83 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 403.20 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 409.93 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 416.00 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 417.34 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 423.97 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 430.77 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 437.41 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 444.19 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 449.11 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 455.84 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 462.59 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 469.28 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 475.94 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 482.67 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 488.00 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 488.58 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 494.62 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 498.53 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 505.19 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 511.84 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 518.58 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 525.31 MiB    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 526.27 MiB* Restarting existing docker container for "cloud-code-minikube" ...
! StartHost failed, but will try again: provision: get ssh host-port: get port 22 for "cloud-code-minikube": docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" cloud-code-minikube: exit status 1
stdout:


stderr:
Template parsing error: template: :1:4: executing "" at <index (index .NetworkSettings.Ports "22/tcp") 0>: error calling index: index of untyped nil

* Stopping "cloud-code-minikube" in docker ...
* Deleting "cloud-code-minikube" in docker ...
* Creating docker container (CPUs=2, Memory=5901MB) ...
* Failed to start docker container. "minikube start -p cloud-code-minikube" may fix it: creating host: create: creating: create kic node: container name "cloud-code-minikube": container exited unexpectedly

! The minikube  container exited unexpectedly.
* If you are still interested to make  driver work. The following suggestions might help you get passed this issue:
* 
	- Prune unused  images, volumes and abandoned containers.
* 
	- Restart your  service
* 
	- Ensure your  daemon has access to enough CPU/memory resources. 
* 
	- Delete and recreate minikube cluster
		minikube delete
		minikube start --driver=
* 
X error provisioning host: Failed to start host: creating host: create: creating: create kic node: container name "cloud-code-minikube": container exited unexpectedly
* 
* minikube is exiting due to an error. If the above message is not useful, open an issue:
  - https://github.com/kubernetes/minikube/issues/new/choose
@tstromberg tstromberg changed the title Minikube with Docker driver and '--delete-on-failure' failed to start docker v1.12: <index (index .NetworkSettings.Ports "22/tcp") 0>: error calling index: index of untyped nil -> container exited unexpectedly Jul 23, 2020
@tstromberg
Copy link
Contributor

The first run fails due to:

Template parsing error: template: :1:4: executing "" at <index (index .NetworkSettings.Ports "22/tcp") 0>: error calling index: index of untyped nil

The second fails due to:

container name "cloud-code-minikube": container exited unexpectedly

@medyagh - do you mind looking into this and make these errors easier to debug? The output of docker info would go a long way for the first error, and the output of docker logs <container> for the second.

@tstromberg tstromberg added co/docker-driver Issues related to kubernetes in container kind/bug Categorizes issue or PR as related to a bug. labels Jul 23, 2020
@tstromberg
Copy link
Contributor

tstromberg commented Jul 23, 2020 •
edited
Loading

I believe some investigative has gone into both of these things in the past, but I also wonder if they are race conditions where we are too impatient or not waiting for the right set of events to occur first.

@tstromberg
Copy link
Contributor

/cc @balopat - he did mention that this may have been upgrade related, but he no longer has logs. It's possible that minikube v1.12.1 may have accidentally fixed the upgrade issue, but I do not have high confidence in that.

I do not believe we have done anything to make the two failures easier to debug, which we should.

@medyagh
Copy link
Member

medyagh commented Jul 23, 2020

this basicly minikube created a container but it shortly exited, and still the advice that minikube gave is relevent.

however in july-chil we added a Pr that checks the Docker Desktop memory size.

one possible thing is, I see Creating docker container (CPUs=2, Memory=5901MB

it could be the docker desktop has only 2 GB and not enough memory.

do you mind sharing how much memory docker desktop has ?

docker > settings > resources ...

@tstromberg
Copy link
Contributor

@balopat mentioned to me that he was upgrading from minikube v1.11.0 on macOS. Let's make sure to test this upgrade case a couple of times before closing this issue.

@tstromberg
Copy link
Contributor

Action items:

  • Replace the Template parsing error: template: :1:4: executing "" at <index (index .NetworkSettings.Ports "22/tcp") 0>: error calling index: index of untyped nil error with something more useful. Like Unable to find SSH port for "minikube" container: docker info`

  • Add docker logs <container> to output of The minikube container exited unexpectedly. message

  • Update advice to be more specific about which actions to take: how does one prune images?

  • Interactively test v1.11.0 -> HEAD upgrade cycle 10 times and assert that no errors are thrown

@medyagh
Copy link
Member

medyagh commented Jul 23, 2020

Action items:

I don't think we should flood the display with verbose logs, when we are sure the reason is docker desktop under resourced.

  • Update advice to be more specific about which actions to take: how does one prune images?
    this one is specific to driver and also comes with some risks, I rather they look it up on their own. in docker docs
  • Interactively test v1.11.0 -> HEAD upgrade cycle 10 times and assert that no errors are thrown

@medyagh
Copy link
Member

medyagh commented Jul 23, 2020

  • Replace the Template parsing error: template: :1:4: executing "" at <index (index .NetworkSettings.Ports "22/tcp") 0>: error calling index: index of untyped nil error with something more useful. Like Unable to find SSH port for "minikube" container: docker info`
    sure we can do that, for the users that have delete-on-failure, so they see a more pleasant error.

@tstromberg
Copy link
Contributor

This may be a dupe of #8179

The output seems to consistently include update-alternatives: error: no alternatives for iptables

This appears to have no relation to under-resourced docker, but is in fact a minikube bug -- which is why the logs would be very helpful.

@tstromberg
Copy link
Contributor

tstromberg commented Jul 23, 2020 •
edited
Loading

I was able to duplicate this with v1.11 -> HEAD using the stress test script found in #8816

Log is attached.

log.txt

@tstromberg
Copy link
Contributor

docker logs showed: update-alternatives: error: no alternatives for iptables

docker info:

Client:
 Debug Mode: false

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 463
 Server Version: 19.03.8
 Storage Driver: overlay2
  Backing Filesystem: <unknown>
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.19.76-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 3.848GiB
 Name: docker-desktop
 ID: LVSK:UU2I:TDYX:E76S:7N5E:IMEZ:6PCO:NVVE:QHO4:EFVO:SU6Z:VUNF
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 43
  Goroutines: 54
  System Time: 2020-07-23T21:16:34.9569881Z
  EventsListeners: 4
 HTTP Proxy: gateway.docker.internal:3128
 HTTPS Proxy: gateway.docker.internal:3129
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

CPU's: 2
Memory: 4GB
Swap: 512 MB
Disk image: 40GB

File sharing is default: /Users /Volumes /private /tmp

This was referenced Jul 24, 2020
Open
Closed
@tstromberg tstromberg added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Jul 27, 2020
@medyagh medyagh mentioned this issue Jul 28, 2020
Merged
@tstromberg
Copy link
Contributor

Fixed at HEAD.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@medyagh medyagh

Labels
co/docker-driver Issues related to kubernetes in container july-chill kind/bug Categorizes issue or PR as related to a bug. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@tstromberg @etanshaul @medyagh