Extra empty sub folders created when using --recursive and --preserve-permissions

[nickjacobsen]

Which version of the AzCopy was used?

Note: The version is visible when running AzCopy without any argument

AzCopy 10.13.0

Which platform are you using? (ex: Windows, Mac, Linux)

Windows

What command did you run?

Note: Please remove the SAS to avoid exposing your credentials. If you cannot remember the exact command, please retrieve it from the beginning of the log file.

azcopy.exe copy "https://.blob.core.windows.net/<source_container>//?" "https://.blob.core.windows.net/<destination_container>/?" --preserve-permissions --recursive ;

What problem was encountered?

The source and destination were two different (ADLS Gen2) Azure Storage Containers
It copies empty folder structure to the destination, in the .
An empty recursive structure is replicated one directory down.

If we have source:

• source directory
  -- File1

Destination has

• source directory
  -- File1
  -- source directory (this is created; and any others in the recursion of the source)

In creating the extra directory it gets permissions errors. that are related to the creation of the extra directory

===== RESPONSE ERROR (ServiceCode=AuthorizationPermissionMismatch) =====
Description=403 This request is not authorized to perform this operation using this permission., Details: (none)
   PUT https://<storageaccount>.dfs.core.windows.net/<destination container>/<directory>?action=setaccesscontrol&se=2021-12-30t05%3A22%3A57z&sig=-REDACTED-&sp=rwl&sr=c&sv=2020-08-04&timeout=901

How can we reproduce the problem in the simplest way?

Run the command between two different contaners.

Have you found a mitigation/solution?

If I leave off the --preserve-permissions it doesn't create the extra directory but I also don't get the permission propagated.

[mohsha-msft]

Hey @nickjacobsen ,

I see that you're trying to transfer files from one blob storage account (ADLS Gen2) to another. The --preserve-permissions flag is intended to preserves ACLs between aware resources (Windows and Azure Files, or ADLS Gen 2 to ADLS Gen 2). This flag applies to both files and folders, unless a file-only filter is specified (e.g. include-pattern).

Are you saying that AzCopy created empty folders which were not present at the source location?
You can share your log file with us to investigate this further if some transfers don't make sense to you.

[nickjacobsen]

It transferred all the files, folders and permissions as expected, however, in addition to the files and folders, it created an empty directory structure replicating the original within the root directory of the original structure.

So in my case I copied a directory which has 4 sub-directories as below

root_directory

• sub_directory1
• sub_directory2
• sub_directory3
• sub_directory4

What I ended up with
root_directory

• sub_directory1
• sub_directory2
• sub_directory3
• sub_directory4
• root_directory (this is where the new ones start, named the same as the root)
  -- sub_directory1 (Empty)
  -- sub_directory2 (Empty)
  -- sub_directory3 (Empty)
  -- sub_directory4 (Empty)

[mohsha-msft]

Hmm, that's strange. Can you please share the log with me. The email is [email protected].

[nickjacobsen]

I've just done a test without using a SAS token for auth to the destination and it still gives the same outcome.

[radoslawcz]

I observed the same behavior. I was going to raise an issue for that but found the issue is already opened for this. The described behavior happens only when you want to copy data between two data lake storage accounts with the "--preserve-permissions" flag added to preserve ACLs, and only when you want to copy a specific directory from a container. When you synchronize whole containers (not a specific directory within a container) the issue does not occur.

[bigdatamoore]

We are running into the same issue. Can someone from Microsoft please take a look at this?

[mohsha-msft]

Fix released in AzCopy 10.14.0