azcopy sync ignoring AZCOPY_BUFFER_GB

[aruvic]

Which version of the AzCopy was used?

Note: The version is visible when running AzCopy without any argument

AzCopy 10.6.0

Which platform are you using? (ex: Windows, Mac, Linux)

Windows

What command did you run?

Note: Please remove the SAS to avoid exposing your credentials. If you cannot remember the exact command, please retrieve it from the beginning of the log file.

azcopy.exe sync %source% %dest% --put-md5 --recursive=true --delete-destination=true --log-level=ERROR

What problem was encountered?

Even though there is AZCOPY_BUFFER_GB=8 and in the azcopy log Max file butter RAM 8.000 GB memory consumption grows up to 20 GB and we need to kill it.

How can we reproduce the problem in the simplest way?

Try to sync a 2TB volume with thousands of folders and millions of files to a Azure Blob

Have you found a mitigation/solution?

No

[amishra-dev]

Hi @aruvic
Unfortunately this is a documentation bug. What this flag does is it specifies the memory that can be used to cache downloads or uploads. So in someway this is the minimum memory used by azcopy, not the maximum as specified by the doc. I will get the document fixed.

[aruvic]

Hi,

Are you sure that this is a documentation issue? In previous issues and release notes this flag was announced as the way to limit the memory consumption.

#802

Br,
Alen

[tianweiliutw]

+1 on this issue. We currently have no way of syncing or copying over 15,000,000 files from one storage account to another (most of them skipped) inside a VM with 8GB of RAM.

[Sylfwood]

+1 on this issue too. It is impossible for us to sync over 7,300,000 files without using at least 10GB of RAM.

[marc-hb]

AZCOPY_BUFFER_GB and other configuration problems aside, azcopy sync memory usage seems to grow really fast and linearly with the number of files. Is there already an issue filed about this memory (ab)use and lack of scalability?

This is surprising and disappointing considering azcopy jobs resume offers the ability to complete a partially failed transfer thanks to the "plans" information on disk (see #221, #693 and others). If all the information needed is on disk then why does azcopy sync need so much RAM? It should feel the need to hold the entire "plan" in RAM at once; there are only so many files that can be transferred at the same time.