Bug 1867152: <carry>: Umount volumes with force #31
Conversation
In 4.5, NFS driver pod used 'clientaddr=<pod ip>' as a mount option. When such pod restarts, it gets a different IP address and volumes mounted by the old pod are basically broken - all operations (stat, umount) will just hang forever. Therefore: 1. Do not use IsLikelyNotMountPoint - it runs stat() on the volume, which could hang forever. Read /etc/mounts instead. 2. Use "umount -f" to forcefully remove NFS mount. Carry at least for whole OCP 4.6, where we need to support move from 4.5.
openshift-ci-robot
added
bugzilla/severity-high
|
@jsafrane: This pull request references Bugzilla bug 1867152, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker. 3 validation(s) were run on this bug
DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
jsafrane
force-pushed
the
fix-umount-force
branch
from
393dfb7 to
3eb3e51
Compare
|
@jsafrane: This pull request references Bugzilla bug 1867152, which is valid. 3 validation(s) were run on this bug
DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
|
||
| for _, mnt := range mtab { | ||
| // This is how a directory deleted on the NFS server looks like | ||
| deletedDir := fmt.Sprintf("%s\\040(deleted)", mnt.Path) |
There was a problem hiding this comment.
When pod's IP address changes - does this "deleted" text appears in mount table? what happens if NFS server is unavailable?
There was a problem hiding this comment.
This text appears when the mounted directory is deleted on the NFS server. It indicates a stale NFS handle.
It's copied from in-tree code, https://github.com/kubernetes/kubernetes/blob/449810c7858cc84e98ff0c859f863a40b56710dc/vendor/k8s.io/utils/mount/mount_helper_unix.go#L155
tombarron
left a comment
tombarron
left a comment
There was a problem hiding this comment.
Unless the in-tree driver suggests a more subtle approach then using this approach until we're confident that the are no more mounts using the pod address rather than the node address makes sense to me.
jsafrane
force-pushed
the
fix-umount-force
branch
from
3eb3e51 to
59fe400
Compare
|
Reworked to try unmount for 1 minute and then use |
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: Fedosin, jsafrane The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci-robot
added
the
approved
|
@jsafrane: All pull requests linked via external trackers have merged: Bugzilla bug 1867152 has been moved to the MODIFIED state. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
* pkg/nfs/nodeserver.go
Significant manual merging due to conflicts with downstream
commit 59fe400: openshift#31
6 participants
In 4.5, NFS driver pod used
'clientaddr=<pod ip>'as a mount option. When such pod restarts, it gets a different IP address and volumes mounted by the old pod are basically broken - all operations (stat,umount, ...) will just hang forever.Therefore:
Do not use
IsLikelyNotMountPoint- it runsstat()on the volume, which could hang forever. Read/etc/mountsviamounter.List()insteadUse
umount -fto forcefully remove NFS mount.Carry at least for whole OCP 4.6, where we need to support upgrade from 4.5.
cc @openshift/storage @Fedosin @tombarron