Private endpoints

[Work in progress] Issue:

Unable to copy data from cross region using AZ copy or storage explorer.

Environment configuration:

Customer has storage account firewall enabled with private endpoint configured for each storage account.

Private endpoint in the same region uses same VNET.

Private endpoint in different region uses different VNET.

Source storage account: sourceAcc

Destination storage account: destAcc

Troubleshooting and findings:

Tried copying data from West Europe to North Europe and vice-versa.

Got an error “The specified resource does not exist”.

Copying data in the same region works as expected.

Tried performing a copy in our lab, able to copy by whitelisting the public IP of the client machine.

Tried whitelisting the client IP on customer storage account however the issue persists.

Customer cannot change the firewall settings due to their internal policy.

Hence, we cannot set the firewall rule to All networks for testing purpose.

We tried copying one file and it also failed.

One of the strange behaviors, we noticed was that if customer copies multiple files, it copies some files and not some.

Reviewed the logs captured, it looks like it is unable to access the storage account of the source.

Since, there is private endpoint in place seems the copy is happening through the private endpoint route and not considering the public IP which we whitelisted.

Researched for known issues about private endpoints for Azure Storage.

Found the article below:

https://learn.microsoft.com/en-us/azure/storage/common/storage-private-endpoints

Storage access constraints for clients in VNets with private endpoints

Clients in VNets with existing private endpoints face constraints when accessing other storage accounts that have private endpoints. For example, suppose a VNet N1 has a private endpoint for a storage account A1 for Blob storage. If storage account A2 has a private endpoint in a VNet N2 for Blob storage, then clients in VNet N1 must also access Blob storage in account A2 using a private endpoint. If storage account A2 does not have any private endpoints for Blob storage, then clients in VNet N1 can access Blob storage in that account without a private endpoint.

This constraint is a result of the DNS changes made when account A2 creates a private endpoint.

If we consider it is hitting the issue due to the constraint above, we are still unable to conclude on why some files are getting copied and some are not.