Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(CSI-344): delete filesystems out-of-band if not waiting for object deletion #477

Open
wants to merge 2 commits into
base: dev
Choose a base branch
from

Conversation

sergeyberezansky
Copy link
Collaborator

@sergeyberezansky sergeyberezansky commented Mar 11, 2025

TL;DR

Improved CSI plugin latency on DeleteVolume by moving the deletion of actual filesystem and NFS permissions to asynchronous flow. This behavior is configurable via pluginConfig.waitForObjectDeletion flag

What changed?

  • Added retry logic (up to 5 attempts with 1-second intervals) when deleting NFS permissions to handle transient failures such as concurrent deletion of multiple entries at once.
  • Added cache invalidation of filesystem object upon getUnderlyingSnapshots
  • Improved error message to indicate when NFS permission deletion fails after multiple attempts

How to test?

  1. Create a volume with NFS transport
  2. Delete the volume and verify that NFS permissions are properly cleaned up

Why make this change?

These changes address latency of DeleteVolume request by moving the deletion into out-of-band thread. The retry logic for NFS permission deletion helps overcome transient API failures that could possibly block filesystem deletion, for example in case of multiple deletions at once that require massive changes to NFS server reconfiguration.

Copy link

graphite-app bot commented Mar 11, 2025

Graphite Automations

"Request reviewers once CI passes" took an action on this PR • (03/11/25)

1 reviewer was added to this PR based on Sergey Berezansky's automation.

@sergeyberezansky sergeyberezansky changed the base branch from sergey/replace-mounts-with-api to graphite-base/477 March 11, 2025 12:39
@sergeyberezansky sergeyberezansky force-pushed the sergey/remove-filesystem-nfs-permissions-out-of-band branch from 7c993f4 to e8f8ebc Compare March 11, 2025 19:55
@sergeyberezansky sergeyberezansky changed the base branch from graphite-base/477 to sergey/replace-mounts-with-api March 11, 2025 19:55
@sergeyberezansky sergeyberezansky force-pushed the sergey/replace-mounts-with-api branch 2 times, most recently from c53f25d to 350def4 Compare March 12, 2025 08:27
@sergeyberezansky sergeyberezansky force-pushed the sergey/remove-filesystem-nfs-permissions-out-of-band branch from 8da3304 to 31f7341 Compare March 12, 2025 08:27
@sergeyberezansky sergeyberezansky force-pushed the sergey/replace-mounts-with-api branch from 350def4 to ff24efe Compare March 12, 2025 10:32
@sergeyberezansky sergeyberezansky force-pushed the sergey/remove-filesystem-nfs-permissions-out-of-band branch from 31f7341 to 2d1fe67 Compare March 12, 2025 10:32
@sergeyberezansky sergeyberezansky force-pushed the sergey/replace-mounts-with-api branch from ff24efe to e551e20 Compare March 12, 2025 12:52
@sergeyberezansky sergeyberezansky force-pushed the sergey/remove-filesystem-nfs-permissions-out-of-band branch from 2d1fe67 to 9695a20 Compare March 12, 2025 12:52
@sergeyberezansky sergeyberezansky force-pushed the sergey/remove-filesystem-nfs-permissions-out-of-band branch from 9695a20 to 809a6e3 Compare March 12, 2025 13:36
@sergeyberezansky sergeyberezansky force-pushed the sergey/replace-mounts-with-api branch 3 times, most recently from 1a1c89e to 5a3a384 Compare March 12, 2025 13:50
@sergeyberezansky sergeyberezansky force-pushed the sergey/remove-filesystem-nfs-permissions-out-of-band branch from 809a6e3 to 7b1bf2b Compare March 12, 2025 13:50
@sergeyberezansky sergeyberezansky changed the base branch from sergey/replace-mounts-with-api to graphite-base/477 March 13, 2025 09:43
@sergeyberezansky sergeyberezansky force-pushed the sergey/remove-filesystem-nfs-permissions-out-of-band branch from 7b1bf2b to 2f33823 Compare March 13, 2025 10:17
@sergeyberezansky sergeyberezansky changed the base branch from graphite-base/477 to sergey/replace-mounts-with-api March 13, 2025 10:18
@sergeyberezansky sergeyberezansky force-pushed the sergey/remove-filesystem-nfs-permissions-out-of-band branch from 2f33823 to 418375b Compare March 13, 2025 10:18
@sergeyberezansky sergeyberezansky changed the base branch from sergey/replace-mounts-with-api to graphite-base/477 March 13, 2025 10:43
@sergeyberezansky sergeyberezansky force-pushed the sergey/remove-filesystem-nfs-permissions-out-of-band branch from 418375b to 4c85fb2 Compare March 13, 2025 10:43
@sergeyberezansky sergeyberezansky changed the base branch from graphite-base/477 to dev March 13, 2025 10:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant