node-agent not waiting for PV removal

[ix-rzi]

What steps did you take and what happened:

The node-agent does not wait for the PV to be removed on the cluster before deleting the VolumeSnapshot. As we are using NetApp Trident and NetApp ONTAP based storage this affects us in the following way:

Most of the time the NetApp FlexClone that is made available as PVC for the data upload is still in a deleting state on the NetApp filer when the VolumeSnapshot is deleted as part of the cleanup. As a consequence, NetApp Trident identifies the snapshot that is the source of the FlexClone as 'busy' and will perform a split clone operation, turning the FlexClone into a real clone.

This means that all the data will be copied into an independent volume and will be deleted afterwards. While the waste of space is temporary, it can take hours to perform this operation and may have the potential to fill up the storage. Trident is not able to remove the volume that is the base for the PVC as long as the clone operation is in progress.

I do not think this is a problem with NetApp Trident!

Commands
velero install --provider aws
--bucket velero
--plugins velero/velero-plugin-for-aws:v1.11.0
--secret-file ./secret-file
--use-volume-snapshots=true
--backup-location-config region="minio",s3ForcePathStyle="true",s3Url=http://xxx.yyy.zzz.aaa:9000
--snapshot-location-config region="minio"
--use-node-agent
--features=EnableCSI
--node-agent-configmap=node-agent-config

velero backup create test01 --include-namespaces nginx --snapshot-move-data

What did you expect to happen:

node-agent waits for the PV to disappear before deleting the VolumeSnapshot.

The following information will help us better understand what's going on:

PoC code to wait for the PV delete. When running the forked node-agent the split clones no longer happen:

main...ix-rzi:velero:fix-ensure-pv-removed

I'm happy to discuss and open a PR.

Environment:

Note, this is a local dev/test environment!

• Velero version : v1.15.2
• Velero features :
• Kubernetes version: v1.31.5+k3s1
• Kubernetes installer & version: K3s
• Cloud provider or hardware configuration: K3s on Fedora 41
• OS (e.g. from /etc/os-release):
  NAME="Fedora Linux"
  VERSION="41 (Server Edition)"
  RELEASE_TYPE=stable
  ID=fedora
  VERSION_ID=41
  VERSION_CODENAME=""
  PLATFORM_ID="platform:f41"
  PRETTY_NAME="Fedora Linux 41 (Server Edition)"
  ANSI_COLOR="0;38;2;60;110;180"
  LOGO=fedora-logo-icon
  CPE_NAME="cpe:/o:fedoraproject:fedora:41"
  HOME_URL="https://fedoraproject.org/"
  DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f41/system-administrators-guide/"
  SUPPORT_URL="https://ask.fedoraproject.org/"
  BUG_REPORT_URL="https://bugzilla.redhat.com/"
  REDHAT_BUGZILLA_PRODUCT="Fedora"
  REDHAT_BUGZILLA_PRODUCT_VERSION=41
  REDHAT_SUPPORT_PRODUCT="Fedora"
  REDHAT_SUPPORT_PRODUCT_VERSION=41
  SUPPORT_END=2025-12-15
  VARIANT="Server Edition"
  VARIANT_ID=server

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• 👍 for "I would like to see this bug fixed as soon as possible"
• 👎 for "There are more important bugs to focus on right now"

[blackpiglet]

Let me try to sort out what happens here.

• The problem happens when Velero deletes the backup or restore Pod, PVC, PV, VS, and VSC.
• The unwanted is that the backup or restore PVC's data source, which is the backup or restore VS, are deleted before the PVC is deleted. That caused the full clone of the PVC volume from the snapshot. I think that is reasonable because the volume source is gone, and the solution should be making the volume independent from the snapshot source.

If my understanding is correct, then the enhancement should modify the backup restore resource cleaning order.
Your modification should work, but I think we should also consider whether there are some error cases that may also have a similar issue.

[Lyndon-Li]

@ix-rzi

PVC for the data upload is still in a deleting state on the NetApp filer when the VolumeSnapshot is deleted as part of the cleanup

Thanks for the good catch and analysis. Please submit the PR and let's discuss in the PR for minor changes.

The changes are based on the fact that the FlexClone has been fully deleted from the storage when the backupPV disappears from the cluster, otherwise, it still doesn't help. But I think Kubernetes and CSI driver should be working in this way and guarantee this.

[Lyndon-Li]

@blackpiglet Let's track the cleanup logics for error handling separately. Probably, for various cases in error handling, make DeletePVAndPVCIfAny do the same wait is a good idea.

[ix-rzi]

@Lyndon-Li Indeed, K8s and the CSI driver should take care of this, assuming a correct CSI driver implementation. I will submit the PR ASAP.