warn about "stuck docker" in minikube delete #6807
Conversation
k8s-ci-robot
added
cncf-cla: yes
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: medyagh The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
the
approved
pkg/drivers/kic/oci/oci.go
Outdated
| // if it doesn't it means docker-daemon is stuck and needs restart | ||
| if err != nil { | ||
| deleteErrs = append(deleteErrs, errors.Wrapf(err, "delete container %s: -daemon is stuck.", c, ociBin)) | ||
| glog.Errorf("%s-daemon seems to be stuck. Please try restarting your %s.", ociBin, ociBin) |
There was a problem hiding this comment.
Won't this result in duplicate errors being sent to the console? In general, you either log an error, or return it, but not both.
There was a problem hiding this comment.
no because we dont log in parent, we try to delete as much as we can without providing deetails what failed..
pkg/drivers/kic/oci/oci.go
Outdated
| cmd := exec.Command(ociBin, "rm", "-f", "-v", c) | ||
| if out, err := cmd.CombinedOutput(); err != nil { | ||
| deleteErrs = append(deleteErrs, errors.Wrapf(err, "delete container %s: output %s", c, out)) | ||
| _, err := containerStatus(ociBin, c) |
There was a problem hiding this comment.
Is there a command available to detect this that does not require querying the status of an individual container?
It seems like it would be better to detect a stuck daemon in a general fashion (like docker status or some other command), rather than one that could be impacted by a container not existing.
There was a problem hiding this comment.
unfortunately that is not the case ! docker status works great ! and it even lets your create new containers
and everything is normal ! only this container is stuck and u can not do anything to it
that was only way I could detect it.
k8s-ci-robot
added
size/L
Codecov Report
@@ Coverage Diff @@
## master #6807 +/- ##
==========================================
- Coverage 38.5% 38.43% -0.08%
==========================================
Files 142 142
Lines 8695 8711 +16
==========================================
Hits 3348 3348
- Misses 4927 4943 +16
Partials 420 420
|
medyagh
changed the title
Before PR:
delete would hang forever...
because docker container was stuck (however docker itself is healthy and no signs of docker problem but in reality docker needs restart, only way to identify this issue is to do docker inspect on the container and it would hang forever ... (no error) and other docker commands like info or system info were just acting normal)
after this PR:
Fail Fast. Warn User to restart docker.
nothing should hang forever.