Race condition happens in shallow copy volume

[saurabhwani5]

Describe the bug

when shallow copy volume and snapshot deleted together then race condition happens and snapshot doesn't get deleted with log error as following
I have uploaded complete test run report here : /u/DUMPS/scale-csi/D.1221

E0926 06:15:36.386636       1 rest_v2.go:162] [0b437ec0-2a66-4d93-922c-647ebd5a251c] Async Job failed: {{200 The request finished successfully.} [{{[] [] 6 [EFSSG0264C The path /var/mnt/local-sample/pvc-4a29738e-01e1-4431-bbd7-d17dbff22080/snapshot-191e5413-29a8-4fd2-997e-8cc8f940b2c4 does not exist.] []} {GET /scalemgmt/v2/filesystems/local-sample/directory/pvc-4a29738e-01e1-4431-bbd7-d17dbff22080%2Fsnapshot-191e5413-29a8-4fd2-997e-8cc8f940b2c4 map[]} 1000000023847 2024-09-26 06:15:34,370 2024-09-26 06:15:34,470 FAILED}]}
E0926 06:15:36.386706       1 controllerserver.go:3129] [0b437ec0-2a66-4d93-922c-647ebd5a251c] unable to stat directory using FS [local-sample] at path [pvc-4a29738e-01e1-4431-bbd7-d17dbff22080/snapshot-191e5413-29a8-4fd2-997e-8cc8f940b2c4]. Error [unable to stat dir pvc-4a29738e-01e1-4431-bbd7-d17dbff22080/snapshot-191e5413-29a8-4fd2-997e-8cc8f940b2c4:[EFSSG0264C The path /var/mnt/local-sample/pvc-4a29738e-01e1-4431-bbd7-d17dbff22080/snapshot-191e5413-29a8-4fd2-997e-8cc8f940b2c4 does not exist.]]

How to Reproduce?

1. PVCs :

    [root@k8s~]# kubectl get pvc -w
NAME             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS    VOLUMEATTRIBUTESCLASS   AGE
pvc-mjltdvgkgw   Bound    pvc-4a29738e-01e1-4431-bbd7-d17dbff22080   1Gi        RWX            sc-htfdiygqaw   <unset>                 75s
restored-pvc-rhckeut-0   Bound     pvc-8847b497-775b-4fac-a192-5fc0d33abd74   1Gi        ROX            sc-htfdiygqaw   <unset>                 7s
restored-pvc-rhckeut-0                      Bound     pvc-8847b497-775b-4fac-a192-5fc0d33abd74   1Gi        ROX            sc-htfdiygqaw   <unset>                 32s
clone-restored-pvc-rhckeut-0-0-0-lfwoizlj   Bound     pvc-03e74d8d-8942-438b-901c-a248817d4ae6   1Gi        RWX            sc-htfdiygqaw   <unset>                 57s
clone-restored-pvc-rhckeut-0-1-0-erkmd      Bound     pvc-41503de7-4256-45e0-8171-203677dc6323   1Gi        RWO            sc-htfdiygqaw   <unset>                 73s

2. VS :

[root@k8s ~]# kubectl get vs -w
NAME           READYTOUSE   SOURCEPVC        SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS      SNAPSHOTCONTENT   CREATIONTIME   AGE
vs-rhckeut-0   true         pvc-mjltdvgkgw                           1Gi           vs-class-dlbkqlr   snapcontent-191e5413-29a8-4fd2-997e-8cc8f940b2c4   4s             6s

Expected behavior

Source PV and snapshot should be deleted

Data Collection and Debugging

CSI snap and failed test case log :
/u/DUMPS/scale-csi/D.1221