Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

IOException when building from mapping pack files into memory #584

Closed
ssa3512 opened this issue Apr 9, 2021 · 19 comments · Fixed by #626
Closed

IOException when building from mapping pack files into memory #584

ssa3512 opened this issue Apr 9, 2021 · 19 comments · Fixed by #626
Labels
bug managed-git
Milestone
v3.4

Comments

@ssa3512
Copy link

ssa3512 commented Apr 9, 2021

Several developers reporting and at least one CI build failed. Rolled back to 3.3.37 for now.

System.IO.IOException: Not enough memory resources are available to process this command.

   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.MemoryMappedFiles.MemoryMappedView.CreateView(SafeMemoryMappedFileHandle memMappedFileHandle, MemoryMappedFileAccess access, Int64 offset, Int64 size)
   at Nerdbank.GitVersioning.ManagedGit.GitPack..ctor(GetObjectFromRepositoryDelegate getObjectFromRepositoryDelegate, String indexPath, String packPath, GitPackCache cache)
   at Nerdbank.GitVersioning.ManagedGit.GitPack..ctor(GitRepository repository, String name)
   at Nerdbank.GitVersioning.ManagedGit.GitRepository.LoadPacks()
   at System.Lazy`1.CreateValue()
   at System.Lazy`1.LazyInitValue()
   at System.Lazy`1.get_Value()
   at Nerdbank.GitVersioning.ManagedGit.GitRepository.TryGetObjectBySha(GitObjectId sha, String objectType, Stream& value)
   at Nerdbank.GitVersioning.ManagedGit.GitRepository.GetObjectBySha(GitObjectId sha, String objectType)
   at Nerdbank.GitVersioning.ManagedGit.GitRepository.GetCommit(GitObjectId sha, Boolean readAuthor)
   at Nerdbank.GitVersioning.ManagedGit.GitRepository.GetHeadCommit(Boolean readAuthor)
   at Nerdbank.GitVersioning.Managed.ManagedGitContext..ctor(String workingDirectory, String dotGitPath, String committish)
   at Nerdbank.GitVersioning.GitContext.Create(String path, String committish, Boolean writable)
   at Nerdbank.GitVersioning.Tasks.GetBuildVersion.ExecuteInner()
   at MSBuildExtensionTask.ContextAwareTask.Execute()
   at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute()
   at Microsoft.Build.BackEnd.TaskBuilder.<ExecuteInstantiatedTask>d__26.MoveNext()
@AArnott
Copy link
Collaborator

AArnott commented Apr 9, 2021

@qmfrederik Could this be because of trying to map a single, very large pack file? Or is this due to a memory leak in the object cache you build up?

@AArnott AArnott added the bug label Apr 9, 2021
@AArnott AArnott added this to the v3.4 milestone Apr 9, 2021
@qmfrederik
Copy link
Contributor

Thanks, for reporting, @ssa3512 . Could you share the size of your pack files (i.e. .git\objects\pack\*.pack)?

Do you know whether MSBuild is running as a 32-bit or 64-bit process?

@ssa3512
Copy link
Author

ssa3512 commented Apr 9, 2021

@qmfrederik here is from the CI server that failed. Working on getting from the users that also reported issues.

The CI build was set to MSBuild x86 - I am assuming this controls what it runs with. As for the users, they are using Visual Studio 2019 16.8 or newer. Is there a setting somewhere that would control 32 vs 64 bit for building?

image

Directory of F:\agent\ADO-1\_work\2\s\.git\objects\pack

04/09/2021  08:00 AM    <DIR>          .
04/09/2021  08:00 AM    <DIR>          ..
02/19/2021  09:00 AM            34,616 pack-030521c82632a3cbca431a40f4201c8763c91cee.idx
02/19/2021  09:00 AM         3,275,714 pack-030521c82632a3cbca431a40f4201c8763c91cee.pack
03/18/2021  08:00 AM            34,224 pack-09713bb4d25f255eefdd25ee53a9965acb71e5ce.idx
03/18/2021  08:00 AM         2,966,240 pack-09713bb4d25f255eefdd25ee53a9965acb71e5ce.pack
02/18/2021  09:00 AM            19,020 pack-0dccb0a4290da5559f7cd6e706accf4867bdd15b.idx
02/18/2021  09:00 AM         1,065,820 pack-0dccb0a4290da5559f7cd6e706accf4867bdd15b.pack
02/01/2021  09:00 AM            45,816 pack-1291fa4212f89bb3bb809d47d6951af9fd6b3baf.idx
02/01/2021  09:00 AM         3,334,810 pack-1291fa4212f89bb3bb809d47d6951af9fd6b3baf.pack
03/25/2021  08:00 AM            44,836 pack-1b9c5580e5b8661519ddf6885a10b22a049be859.idx
03/25/2021  08:00 AM         3,419,913 pack-1b9c5580e5b8661519ddf6885a10b22a049be859.pack
04/08/2021  08:00 AM            47,832 pack-251aef105561f2856da25e4577018d30b459b769.idx
04/08/2021  08:00 AM         4,338,411 pack-251aef105561f2856da25e4577018d30b459b769.pack
03/24/2021  08:00 AM            41,896 pack-25d345c594df14fca6188fdab584ab594e154bae.idx
03/24/2021  08:00 AM         3,151,989 pack-25d345c594df14fca6188fdab584ab594e154bae.pack
02/15/2021  09:00 AM            19,692 pack-2707cb44f800b30aa9ce30e727a3b759c1d6d308.idx
02/15/2021  09:00 AM         1,512,454 pack-2707cb44f800b30aa9ce30e727a3b759c1d6d308.pack
03/15/2021  08:00 AM            33,244 pack-2e6350ad2dbf91e7ed7e2a6a06c05a7e86f8c8f0.idx
03/15/2021  08:00 AM         2,766,538 pack-2e6350ad2dbf91e7ed7e2a6a06c05a7e86f8c8f0.pack
01/26/2021  09:00 AM            37,892 pack-3050b1273d645f82f3c9a08e4783e976bde9eaba.idx
01/26/2021  09:00 AM         3,044,022 pack-3050b1273d645f82f3c9a08e4783e976bde9eaba.pack
02/23/2021  09:00 AM            41,476 pack-342ec42a0677c9f8c9a42058d4f827f1b312dc02.idx
02/23/2021  09:00 AM         2,906,868 pack-342ec42a0677c9f8c9a42058d4f827f1b312dc02.pack
01/25/2021  09:02 AM        17,888,648 pack-381bb6ec83ab07db872d3c63d9836885fe3db4c6.idx
01/25/2021  09:02 AM       363,090,944 pack-381bb6ec83ab07db872d3c63d9836885fe3db4c6.pack
03/11/2021  09:00 AM            47,468 pack-3902fc024943f78be73623dd925b518381a06499.idx
03/11/2021  09:00 AM         2,721,750 pack-3902fc024943f78be73623dd925b518381a06499.pack
03/17/2021  08:00 AM            38,984 pack-3de378f9f4558260c8577ee15535a601fb0c3047.idx
03/17/2021  08:00 AM         2,839,087 pack-3de378f9f4558260c8577ee15535a601fb0c3047.pack
04/05/2021  08:00 AM            16,472 pack-3f69c5f9fbb834a62506b565903a156671e895e7.idx
04/05/2021  08:00 AM         1,333,196 pack-3f69c5f9fbb834a62506b565903a156671e895e7.pack
03/01/2021  09:00 AM            30,500 pack-4034dcc023628cd8c6da65cac72753e4d9d995bb.idx
03/01/2021  09:00 AM         2,528,528 pack-4034dcc023628cd8c6da65cac72753e4d9d995bb.pack
03/02/2021  09:00 AM            26,048 pack-40ee7da667b36f6d3311fe9e1197e52b5db742cd.idx
03/02/2021  09:00 AM         1,675,770 pack-40ee7da667b36f6d3311fe9e1197e52b5db742cd.pack
03/23/2021  08:00 AM            35,288 pack-450fb1522a8548238ee13147b8bece661d896ebe.idx
03/23/2021  08:00 AM         2,307,081 pack-450fb1522a8548238ee13147b8bece661d896ebe.pack
02/12/2021  09:00 AM            29,520 pack-46e3988115bca08baf3a5aa0edb59788b3cea39b.idx
02/12/2021  09:00 AM         2,991,780 pack-46e3988115bca08baf3a5aa0edb59788b3cea39b.pack
03/19/2021  08:00 AM            35,148 pack-4794bbf4e8dc4128fa7770ae50cc4d1d716d45cf.idx
03/19/2021  08:00 AM         2,823,373 pack-4794bbf4e8dc4128fa7770ae50cc4d1d716d45cf.pack
03/29/2021  08:00 AM            18,404 pack-4dbfcfbd7858bc2f9e9aabb989feb82ec8c2195b.idx
03/29/2021  08:00 AM         1,571,299 pack-4dbfcfbd7858bc2f9e9aabb989feb82ec8c2195b.pack
03/22/2021  08:00 AM            30,248 pack-5102ca857646f4a824b65cbe96ae2334ef2a13b7.idx
03/22/2021  08:00 AM         3,021,428 pack-5102ca857646f4a824b65cbe96ae2334ef2a13b7.pack
04/09/2021  08:00 AM            24,984 pack-5202dbd373cda37ee88b95edca9909f8d939d820.idx
04/09/2021  08:00 AM         2,102,816 pack-5202dbd373cda37ee88b95edca9909f8d939d820.pack
02/25/2021  09:00 AM            38,592 pack-5c735cc1bd0c219384b8aae19b4230b89c9f0527.idx
02/25/2021  09:00 AM         3,310,911 pack-5c735cc1bd0c219384b8aae19b4230b89c9f0527.pack
02/10/2021  09:00 AM            29,912 pack-63dcac36c9bd94b534394f3353806787d2116b19.idx
02/10/2021  09:00 AM         2,254,670 pack-63dcac36c9bd94b534394f3353806787d2116b19.pack
02/09/2021  09:00 AM            38,228 pack-64ab946a4f3e94b9bd811316be99aea2d2c52562.idx
02/09/2021  09:00 AM         2,918,755 pack-64ab946a4f3e94b9bd811316be99aea2d2c52562.pack
02/03/2021  09:00 AM            29,268 pack-6699d0b55579b22a9614f1a3437d56504fda8cb7.idx
02/03/2021  09:00 AM         2,380,234 pack-6699d0b55579b22a9614f1a3437d56504fda8cb7.pack
02/11/2021  09:00 AM            26,636 pack-6953e37bb4ce0fdb21502a195331a3c2bf713da0.idx
02/11/2021  09:00 AM         2,359,121 pack-6953e37bb4ce0fdb21502a195331a3c2bf713da0.pack
02/05/2021  09:00 AM            51,052 pack-69ddea155fda22c995d50304429106178eb53d18.idx
02/05/2021  09:00 AM         3,972,743 pack-69ddea155fda22c995d50304429106178eb53d18.pack
03/04/2021  09:00 AM            13,812 pack-6b2510e40508fe7aaabff790182d8dc3b0b96a0a.idx
03/04/2021  09:00 AM         1,391,647 pack-6b2510e40508fe7aaabff790182d8dc3b0b96a0a.pack
01/27/2021  09:00 AM            24,340 pack-719dceeb3b6668f90ea180164e7e75454b42a4e8.idx
01/27/2021  09:00 AM         2,211,789 pack-719dceeb3b6668f90ea180164e7e75454b42a4e8.pack
02/26/2021  09:00 AM            24,844 pack-8449c5b8b7bffde1a2c7a52e0c49b93541879380.idx
02/26/2021  09:00 AM         1,851,742 pack-8449c5b8b7bffde1a2c7a52e0c49b93541879380.pack
03/16/2021  08:01 AM            27,952 pack-887613fee2cf1844843fd7ec29a2c51faea4c5e7.idx
03/16/2021  08:01 AM         2,055,706 pack-887613fee2cf1844843fd7ec29a2c51faea4c5e7.pack
03/03/2021  09:00 AM            33,468 pack-8c01a5d1eb2f3b3c24cde8254d9c1656316f0615.idx
03/03/2021  09:00 AM         2,648,884 pack-8c01a5d1eb2f3b3c24cde8254d9c1656316f0615.pack
03/08/2021  09:00 AM            30,668 pack-917c73c3025a3b5e03b58c2bd5b7fdcedb613747.idx
03/08/2021  09:00 AM         2,778,181 pack-917c73c3025a3b5e03b58c2bd5b7fdcedb613747.pack
02/08/2021  09:00 AM            36,436 pack-97b842e62c6ab7b6128d0e8efab2a62d4ef710f7.idx
02/08/2021  09:00 AM         2,809,333 pack-97b842e62c6ab7b6128d0e8efab2a62d4ef710f7.pack
02/22/2021  09:00 AM            28,008 pack-9ae6d83a3ce8fb35a4ccd6e4629811d2e0aa317f.idx
02/22/2021  09:00 AM         2,272,214 pack-9ae6d83a3ce8fb35a4ccd6e4629811d2e0aa317f.pack
01/29/2021  09:00 AM           103,776 pack-a0ad6559e73a36d87315ff5ccea38bfbea3aef6f.idx
01/29/2021  09:00 AM         4,836,028 pack-a0ad6559e73a36d87315ff5ccea38bfbea3aef6f.pack
04/01/2021  08:00 AM            24,340 pack-ae500605699950842a0162b339faaa11fc729adc.idx
04/01/2021  08:00 AM         2,605,064 pack-ae500605699950842a0162b339faaa11fc729adc.pack
02/17/2021  09:00 AM            34,868 pack-b0af955c1bbae503d4e98f3c3483254b1db06465.idx
02/17/2021  09:00 AM       129,425,975 pack-b0af955c1bbae503d4e98f3c3483254b1db06465.pack
02/24/2021  09:00 AM            52,424 pack-b63177bcce972f68047f1d6a88ab9311551fd836.idx
02/24/2021  09:00 AM         3,128,224 pack-b63177bcce972f68047f1d6a88ab9311551fd836.pack
04/02/2021  08:01 AM            22,548 pack-b7b5ca6774b6903dfb54ab0378b0460258ef4009.idx
04/02/2021  08:01 AM         2,262,420 pack-b7b5ca6774b6903dfb54ab0378b0460258ef4009.pack
03/30/2021  08:00 AM            36,576 pack-ba01731671b772cc2aa8d829c8f3e404e5ede74b.idx
03/30/2021  08:00 AM         2,195,260 pack-ba01731671b772cc2aa8d829c8f3e404e5ede74b.pack
01/28/2021  09:00 AM            23,976 pack-c4bd5865a2e6315c62a7cc877ce7b977025fc179.idx
01/28/2021  09:00 AM         1,826,227 pack-c4bd5865a2e6315c62a7cc877ce7b977025fc179.pack
03/05/2021  09:00 AM            38,032 pack-cc70e5922ae2f4240f23f7c12d8fadd03c8bbf96.idx
03/05/2021  09:00 AM         2,080,619 pack-cc70e5922ae2f4240f23f7c12d8fadd03c8bbf96.pack
03/09/2021  09:00 AM            33,300 pack-cc9added0012a5c61061898fdb3d18900a455df4.idx
03/09/2021  09:00 AM         3,061,010 pack-cc9added0012a5c61061898fdb3d18900a455df4.pack
02/02/2021  09:00 AM            19,552 pack-db3eeb8229b4316dcfd97dc5729d139fd72f722e.idx
02/02/2021  09:00 AM         2,158,313 pack-db3eeb8229b4316dcfd97dc5729d139fd72f722e.pack
02/16/2021  09:00 AM            21,708 pack-df766599352e2992655a60e07c93123de5e61a0e.idx
02/16/2021  09:00 AM         1,797,674 pack-df766599352e2992655a60e07c93123de5e61a0e.pack
03/12/2021  09:00 AM            23,052 pack-e80d3505be4f0024b6efb1d80b7dc1295f039250.idx
03/12/2021  09:00 AM         1,872,492 pack-e80d3505be4f0024b6efb1d80b7dc1295f039250.pack
03/26/2021  08:00 AM            24,760 pack-e831d3bd0d3a06b91124b06c5c6a49a9e0df24cb.idx
03/26/2021  08:00 AM         2,014,086 pack-e831d3bd0d3a06b91124b06c5c6a49a9e0df24cb.pack
03/31/2021  08:00 AM            38,480 pack-eafdee9892e7660f52f4c14863c34fe6cbdd6644.idx
03/31/2021  08:00 AM         2,497,242 pack-eafdee9892e7660f52f4c14863c34fe6cbdd6644.pack
02/04/2021  09:00 AM            27,308 pack-ec92ef7230771854230a970de1d6ca6e47c6e8ba.idx
02/04/2021  09:00 AM         1,925,410 pack-ec92ef7230771854230a970de1d6ca6e47c6e8ba.pack
03/10/2021  09:00 AM            37,192 pack-f6bb304d7cdf2a49c6cfc077c73111e1fbfffa06.idx
03/10/2021  09:00 AM         3,415,778 pack-f6bb304d7cdf2a49c6cfc077c73111e1fbfffa06.pack
04/06/2021  08:00 AM            20,700 pack-f8f4eb0eb2282c49c32f5f95aceb8b55dfdd7f78.idx
04/06/2021  08:00 AM         2,190,479 pack-f8f4eb0eb2282c49c32f5f95aceb8b55dfdd7f78.pack
             108 File(s)    644,936,126 bytes
               2 Dir(s)  511,256,244,224 bytes free

@ssa3512
Copy link
Author

ssa3512 commented Apr 9, 2021

Here's one from a user that was getting the error in Visual Studio.

Another giant directory listing 
04/09/2021  07:59 AM              .
04/09/2021  07:59 AM              ..
03/30/2021  09:14 AM            40,412 pack-01803f62f88707186aa464f19881e52d01df35a2.idx
03/30/2021  09:14 AM         2,392,047 pack-01803f62f88707186aa464f19881e52d01df35a2.pack
03/19/2021  11:22 AM            23,024 pack-05dfd19c5ae32c399376322121f6ba1a69e04756.idx
03/19/2021  12:51 PM         1,686,001 pack-05dfd19c5ae32c399376322121f6ba1a69e04756.pack
08/11/2020  06:49 AM            35,960 pack-076e7bac8432438b685027efd843bccf98c4d9d4.idx
08/11/2020  06:49 AM         3,012,101 pack-076e7bac8432438b685027efd843bccf98c4d9d4.pack
03/17/2021  08:08 AM            29,380 pack-0abf19dc303bcd17da481e5c198eb65ad0dbb41c.idx
03/17/2021  08:28 AM         2,522,447 pack-0abf19dc303bcd17da481e5c198eb65ad0dbb41c.pack
09/21/2020  01:56 PM            13,896 pack-0c1b205912662991bcde09e2300893ab8810badc.idx
09/21/2020  01:56 PM         1,257,200 pack-0c1b205912662991bcde09e2300893ab8810badc.pack
11/20/2020  01:15 PM            58,332 pack-0d90ec3556098845284b7458c998947ee4676025.idx
03/17/2021  08:28 AM         4,614,980 pack-0d90ec3556098845284b7458c998947ee4676025.pack
02/25/2021  06:53 PM            40,384 pack-0fdefdd3a326fb65a942f72658b3a08f28fdf30c.idx
02/25/2021  06:53 PM         3,148,512 pack-0fdefdd3a326fb65a942f72658b3a08f28fdf30c.pack
03/16/2021  07:58 AM            16,220 pack-109342b8d9e9d0c48e7fc6960583a7236267720b.idx
03/17/2021  08:28 AM           804,283 pack-109342b8d9e9d0c48e7fc6960583a7236267720b.pack
07/28/2020  10:35 AM            20,420 pack-1122f515b6a2dd53a6565a3a53ed1dc1cb356c96.idx
07/28/2020  10:35 AM         1,828,842 pack-1122f515b6a2dd53a6565a3a53ed1dc1cb356c96.pack
01/06/2021  02:40 PM            49,792 pack-13ec7aa5dc8a8d4155438947b418348ba87acff6.idx
01/06/2021  02:40 PM         3,674,454 pack-13ec7aa5dc8a8d4155438947b418348ba87acff6.pack
07/24/2020  08:01 AM            25,880 pack-14cec300e3548035694c1d3fa0168139624ca18a.idx
07/24/2020  04:57 PM         1,314,327 pack-14cec300e3548035694c1d3fa0168139624ca18a.pack
10/13/2020  03:50 PM            21,904 pack-164e975e1b10219758c275a2e7e8af2b798a6ae5.idx
10/13/2020  03:50 PM         1,086,486 pack-164e975e1b10219758c275a2e7e8af2b798a6ae5.pack
09/21/2020  05:03 PM             7,736 pack-1c24214daa4519574a6bf0ea3942c7e3518d50ab.idx
09/21/2020  05:03 PM           713,043 pack-1c24214daa4519574a6bf0ea3942c7e3518d50ab.pack
12/18/2020  12:02 PM           168,316 pack-1c3108ca7afaa538e9d8f1509a031b4f2a2236dc.idx
12/18/2020  12:02 PM        13,118,570 pack-1c3108ca7afaa538e9d8f1509a031b4f2a2236dc.pack
02/19/2021  03:25 PM            54,020 pack-1e3e4b2d62d48be2331610b4557def59b03f9216.idx
02/19/2021  03:38 PM         5,013,214 pack-1e3e4b2d62d48be2331610b4557def59b03f9216.pack
09/17/2020  11:55 AM            26,944 pack-1f9d771e1168344f013a87b034d49308c00b6be0.idx
09/17/2020  11:55 AM         2,493,598 pack-1f9d771e1168344f013a87b034d49308c00b6be0.pack
08/13/2020  07:28 AM            17,900 pack-218e337f36c1807768bdab850e336363b24ae7df.idx
08/13/2020  07:28 AM         1,733,500 pack-218e337f36c1807768bdab850e336363b24ae7df.pack
10/21/2020  06:50 AM            25,992 pack-21ccce6e248e7be17f21a3f0c9e3cf6b6eb2629f.idx
10/21/2020  06:50 AM         1,842,945 pack-21ccce6e248e7be17f21a3f0c9e3cf6b6eb2629f.pack
08/18/2020  05:55 PM            23,444 pack-21e2b8cf2396ee2eefa4bb841c6d1535f6a2c7d4.idx
08/18/2020  05:55 PM         1,153,609 pack-21e2b8cf2396ee2eefa4bb841c6d1535f6a2c7d4.pack
12/01/2020  08:10 AM            90,448 pack-21f6633d78dd8717c8101d17160281fb978772f5.idx
12/01/2020  10:07 AM         5,981,766 pack-21f6633d78dd8717c8101d17160281fb978772f5.pack
03/26/2021  04:29 PM             7,176 pack-24220745fb0f29005b3160f926864689308204f1.idx
03/26/2021  04:29 PM           653,001 pack-24220745fb0f29005b3160f926864689308204f1.pack
08/03/2020  04:12 PM            54,860 pack-24737f6dacfb2612980e2dd7b01d697551c98303.idx
08/03/2020  04:12 PM         1,707,387 pack-24737f6dacfb2612980e2dd7b01d697551c98303.pack
08/25/2020  10:13 AM            19,216 pack-267ec64f7309ce5b08f1ff62630ca48520605935.idx
08/25/2020  10:13 AM         1,507,367 pack-267ec64f7309ce5b08f1ff62630ca48520605935.pack
03/26/2021  12:42 PM             6,112 pack-28bc0d26f4d8867c7978c2d83eb908dcd1883f60.idx
03/26/2021  01:18 PM           268,585 pack-28bc0d26f4d8867c7978c2d83eb908dcd1883f60.pack
12/07/2020  03:53 PM            86,892 pack-2ab8a046e1d7acdb8373ff8de43ed4af8be2d7cd.idx
12/07/2020  03:53 PM         5,194,831 pack-2ab8a046e1d7acdb8373ff8de43ed4af8be2d7cd.pack
10/14/2020  08:36 AM            12,132 pack-35aebaad2b76d2562b0e3bb07172aa8c5899ea2b.idx
10/14/2020  08:36 AM           739,242 pack-35aebaad2b76d2562b0e3bb07172aa8c5899ea2b.pack
03/08/2021  06:07 PM            14,764 pack-35d8ecf49e89fc7c73550e52032301ce2c68bed9.idx
03/09/2021  01:27 PM         1,415,446 pack-35d8ecf49e89fc7c73550e52032301ce2c68bed9.pack
08/28/2020  12:02 PM            22,408 pack-36783a86394a4531edce1a78745af85369abef33.idx
08/28/2020  12:02 PM         1,571,347 pack-36783a86394a4531edce1a78745af85369abef33.pack
03/25/2021  07:02 PM            60,936 pack-36da8d8aa8e0a16056405c45467d970db26fac4e.idx
03/26/2021  01:18 PM         4,452,828 pack-36da8d8aa8e0a16056405c45467d970db26fac4e.pack
07/20/2020  04:37 PM            15,184 pack-390b3b99b4011f328c57977b074429465add87e5.idx
07/20/2020  04:38 PM         1,450,479 pack-390b3b99b4011f328c57977b074429465add87e5.pack
01/13/2021  10:56 AM            40,468 pack-3a4e2d94e54bc992526d0896deb09222b9d7b720.idx
01/13/2021  10:56 AM         2,749,804 pack-3a4e2d94e54bc992526d0896deb09222b9d7b720.pack
01/07/2021  01:31 PM            22,744 pack-3c832e155a33357d3a2ad8671e1c6964c3756596.idx
01/07/2021  01:31 PM         1,918,367 pack-3c832e155a33357d3a2ad8671e1c6964c3756596.pack
03/18/2021  03:33 PM            36,184 pack-418807ef74e23f53ae29f0c956f1ea3a122b7b1b.idx
03/23/2021  12:11 PM         3,052,910 pack-418807ef74e23f53ae29f0c956f1ea3a122b7b1b.pack
07/31/2020  05:53 PM            14,092 pack-437886dd2ef76e91bcec305a9c231f0e50587f92.idx
07/31/2020  06:00 PM         1,034,333 pack-437886dd2ef76e91bcec305a9c231f0e50587f92.pack
01/26/2021  01:09 PM            62,672 pack-45daa4a5a4cbb1369a174882651343ffab02f00a.idx
01/26/2021  01:09 PM         4,918,244 pack-45daa4a5a4cbb1369a174882651343ffab02f00a.pack
09/29/2020  07:31 AM            37,724 pack-46ce1df128cf0eeccb07f3348ddd38ab3e66981b.idx
10/01/2020  12:09 PM         3,347,366 pack-46ce1df128cf0eeccb07f3348ddd38ab3e66981b.pack
07/24/2020  04:57 PM             7,708 pack-47dcfc1e725a2c46a588f49f320ea84d02bf8443.idx
07/25/2020  08:02 AM           875,562 pack-47dcfc1e725a2c46a588f49f320ea84d02bf8443.pack
10/15/2020  04:09 PM            46,572 pack-48caba21f21774b73054046c91998f543a622083.idx
10/15/2020  04:09 PM         2,976,098 pack-48caba21f21774b73054046c91998f543a622083.pack
08/27/2020  04:57 PM            26,076 pack-4a767ebf0b4a0066a168843cd4ab14f3293ae234.idx
08/27/2020  04:57 PM         1,851,859 pack-4a767ebf0b4a0066a168843cd4ab14f3293ae234.pack
09/25/2020  07:32 AM            21,204 pack-4aa2168b1c12c168a5c44585406e4efb94cb1c74.idx
03/17/2021  08:28 AM         1,960,662 pack-4aa2168b1c12c168a5c44585406e4efb94cb1c74.pack
01/27/2021  02:51 PM            27,028 pack-4b60fd8a92adb273242d16398a526990568c6987.idx
01/27/2021  02:51 PM         2,242,478 pack-4b60fd8a92adb273242d16398a526990568c6987.pack
02/26/2021  12:19 PM            17,452 pack-4c759da68ca85c069b67e1c7610ff0c382bc72cc.idx
02/26/2021  12:19 PM         1,515,288 pack-4c759da68ca85c069b67e1c7610ff0c382bc72cc.pack
03/12/2021  12:21 PM            10,452 pack-4d07282a0a2cff2b5e3019299f1e4cb364f42203.idx
03/12/2021  12:21 PM           885,366 pack-4d07282a0a2cff2b5e3019299f1e4cb364f42203.pack
03/24/2021  07:50 AM            56,064 pack-4e568b945cc191bd5690bb0f7211e30a48cc87da.idx
03/24/2021  07:50 AM         4,036,885 pack-4e568b945cc191bd5690bb0f7211e30a48cc87da.pack
10/08/2020  02:24 PM            44,332 pack-4f079f217bbdbf45341805afb6c219c15ba04fc3.idx
10/08/2020  02:24 PM         2,945,031 pack-4f079f217bbdbf45341805afb6c219c15ba04fc3.pack
07/21/2020  01:58 PM            12,440 pack-517e3ece0b8f92a68dea7c8c5a5db3e523563c3f.idx
07/23/2020  11:16 AM           984,830 pack-517e3ece0b8f92a68dea7c8c5a5db3e523563c3f.pack
09/24/2020  11:30 AM            16,444 pack-531210c9d65b90dcdf6c48441d2723210ea736fe.idx
09/24/2020  11:30 AM         1,417,287 pack-531210c9d65b90dcdf6c48441d2723210ea736fe.pack
11/18/2020  08:54 AM            64,632 pack-551690ea206190e54f0dea63c3a5d7b2d9647b6b.idx
11/23/2020  11:56 AM         4,234,456 pack-551690ea206190e54f0dea63c3a5d7b2d9647b6b.pack
02/24/2021  12:10 PM            33,048 pack-562b62fa31283c077c960a7e874ba17d3b7d8d93.idx
03/04/2021  08:46 PM         2,498,164 pack-562b62fa31283c077c960a7e874ba17d3b7d8d93.pack
03/10/2021  12:46 PM            47,664 pack-580e0c1bcad7919135d92f28150a2620db7bc049.idx
03/11/2021  12:44 PM         3,107,270 pack-580e0c1bcad7919135d92f28150a2620db7bc049.pack
11/03/2020  02:45 PM           150,480 pack-586f0c4434105004e263ce20c0ef9a675f92248c.idx
11/03/2020  03:05 PM        10,636,384 pack-586f0c4434105004e263ce20c0ef9a675f92248c.pack
08/07/2020  11:21 AM            14,316 pack-5b21f0ead4c6581723eba3e73bd672b31b4e5c4f.idx
08/07/2020  11:21 AM         1,040,564 pack-5b21f0ead4c6581723eba3e73bd672b31b4e5c4f.pack
12/08/2020  05:55 PM            28,120 pack-5bffe1875f15dda3b0697e2af35cb18431df9ebe.idx
12/08/2020  05:55 PM         2,325,476 pack-5bffe1875f15dda3b0697e2af35cb18431df9ebe.pack
01/15/2021  02:22 PM            94,956 pack-5c6279cc90de80f404381a2ce75a7996f7a317dc.idx
01/15/2021  02:22 PM         4,867,561 pack-5c6279cc90de80f404381a2ce75a7996f7a317dc.pack
09/22/2020  06:59 AM             5,636 pack-5d2ec3c137099f689d6efe9fb918b28c725c3a15.idx
09/22/2020  06:59 AM           655,868 pack-5d2ec3c137099f689d6efe9fb918b28c725c3a15.pack
09/10/2020  12:13 PM            37,192 pack-61177358fceeaa89d911edf5e2f89b940ad1ffd7.idx
09/10/2020  12:13 PM         2,720,069 pack-61177358fceeaa89d911edf5e2f89b940ad1ffd7.pack
07/22/2020  03:20 PM            19,664 pack-641bffd8a3217ad9d4a505ff0fdb41c9ae633c5a.idx
07/24/2020  04:57 PM         1,950,678 pack-641bffd8a3217ad9d4a505ff0fdb41c9ae633c5a.pack
03/04/2021  07:20 PM            26,524 pack-64e20846183cb0e83c10962bd0f57130a575f225.idx
03/04/2021  08:46 PM         1,366,912 pack-64e20846183cb0e83c10962bd0f57130a575f225.pack
11/06/2020  12:37 PM            21,064 pack-659818b2b9c3f2e8c64c8eadb528f209972a4cce.idx
11/06/2020  12:37 PM         1,868,674 pack-659818b2b9c3f2e8c64c8eadb528f209972a4cce.pack
08/12/2020  11:11 AM            31,116 pack-6601e52cd2b879367aa40e2fef34b84e2e897ef0.idx
08/12/2020  11:11 AM         2,314,262 pack-6601e52cd2b879367aa40e2fef34b84e2e897ef0.pack
10/13/2020  11:06 AM            27,756 pack-692029b8f164eea0c3610d46da511a6bb17cfccf.idx
10/13/2020  11:06 AM         1,873,168 pack-692029b8f164eea0c3610d46da511a6bb17cfccf.pack
09/03/2020  02:01 PM            11,572 pack-6db1165d5872604a09bb65868ec22175d516b33e.idx
09/03/2020  02:01 PM           742,263 pack-6db1165d5872604a09bb65868ec22175d516b33e.pack
01/18/2021  12:12 PM            45,172 pack-6e3a0762b861db289a8a858c6490217fc3f50965.idx
01/18/2021  12:12 PM         4,055,257 pack-6e3a0762b861db289a8a858c6490217fc3f50965.pack
01/22/2021  09:12 AM            53,852 pack-6ee9be9eb1658896a45aa1dfcedc0ce04e6359eb.idx
01/22/2021  09:12 AM         5,039,255 pack-6ee9be9eb1658896a45aa1dfcedc0ce04e6359eb.pack
07/23/2020  11:07 AM            12,272 pack-6ff510f6b0d88cfb96e5cd63161f10d5e0589a39.idx
07/23/2020  11:16 AM           420,320 pack-6ff510f6b0d88cfb96e5cd63161f10d5e0589a39.pack
07/21/2020  09:22 AM            11,908 pack-74700b7b4d0ec598d32a8aa1f020ce6907f7b5a8.idx
07/23/2020  11:16 AM           931,899 pack-74700b7b4d0ec598d32a8aa1f020ce6907f7b5a8.pack
03/26/2021  09:13 AM             7,624 pack-74b77c568b3a464d7287c9bdb30fb6e0dd25029e.idx
03/26/2021  09:13 AM           738,358 pack-74b77c568b3a464d7287c9bdb30fb6e0dd25029e.pack
10/07/2020  07:27 AM            29,856 pack-77542c7411b49580eb5df85fe384f2939e87103c.idx
10/07/2020  07:27 AM         2,478,628 pack-77542c7411b49580eb5df85fe384f2939e87103c.pack
02/03/2021  06:20 PM            39,768 pack-7787890a91d4c39781a1aeb1f573cef1e0532d24.idx
02/03/2021  06:20 PM         3,021,186 pack-7787890a91d4c39781a1aeb1f573cef1e0532d24.pack
07/28/2020  01:29 PM             6,224 pack-77d2593ae87de649a9ad3ceb176e88cf24e8e666.idx
07/28/2020  01:29 PM           585,497 pack-77d2593ae87de649a9ad3ceb176e88cf24e8e666.pack
02/17/2021  08:13 AM            88,852 pack-7a78e417c1346d9b49903bf9a65d84d66a659fac.idx
03/23/2021  12:11 PM       133,721,058 pack-7a78e417c1346d9b49903bf9a65d84d66a659fac.pack
07/29/2020  07:17 AM            19,104 pack-7b6491d55f3170016dcafb64efc5f3bbc8d4d3ea.idx
07/30/2020  06:43 PM         1,721,476 pack-7b6491d55f3170016dcafb64efc5f3bbc8d4d3ea.pack
04/08/2021  10:25 AM            30,192 pack-7d3abe2ca8d4354bd45269c6da459360ba752724.idx
04/08/2021  10:25 AM         2,721,669 pack-7d3abe2ca8d4354bd45269c6da459360ba752724.pack
04/05/2021  06:21 PM            15,968 pack-7dc59c2888a89b051965c34542087ae39a7851b9.idx
04/05/2021  06:21 PM         1,507,448 pack-7dc59c2888a89b051965c34542087ae39a7851b9.pack
02/02/2021  08:29 AM           150,648 pack-7e6e62288276d60aa8c219fc536c9dbf720af914.idx
02/02/2021  08:29 AM         8,412,479 pack-7e6e62288276d60aa8c219fc536c9dbf720af914.pack
09/21/2020  07:41 AM            49,316 pack-825f5352dd97d6e0809bf1902d4b7b7bba74eb33.idx
09/21/2020  07:41 AM         3,724,871 pack-825f5352dd97d6e0809bf1902d4b7b7bba74eb33.pack
07/22/2020  05:04 PM            10,956 pack-85c1d972a5e3dbc0661cd10ef31b96d5eb23c4bb.idx
07/24/2020  04:57 PM           625,519 pack-85c1d972a5e3dbc0661cd10ef31b96d5eb23c4bb.pack
10/09/2020  06:31 PM            28,260 pack-888ea11200f695e0bbd9ef6ad6a4bf527e83a686.idx
10/09/2020  06:31 PM         3,019,069 pack-888ea11200f695e0bbd9ef6ad6a4bf527e83a686.pack
03/04/2021  11:56 AM            12,944 pack-8a3bc4ce8c28943e414a4266ec3f0e8e8dd8d46d.idx
03/04/2021  08:46 PM         1,292,359 pack-8a3bc4ce8c28943e414a4266ec3f0e8e8dd8d46d.pack
07/30/2020  02:58 PM            25,320 pack-8b3866d7edf0412a13305acd26d5599f5259318c.idx
07/31/2020  06:00 PM         1,781,236 pack-8b3866d7edf0412a13305acd26d5599f5259318c.pack
04/07/2021  09:46 AM            38,396 pack-900eb27c5f6c94f766285a567f4c182f62ed3b1f.idx
04/07/2021  09:46 AM         3,926,534 pack-900eb27c5f6c94f766285a567f4c182f62ed3b1f.pack
02/23/2021  09:05 AM            41,980 pack-912ed8a45a2b3cc2b17b0423313dde04e95ff4ac.idx
02/23/2021  09:05 AM         2,869,274 pack-912ed8a45a2b3cc2b17b0423313dde04e95ff4ac.pack
03/15/2021  02:44 PM            45,564 pack-91bb1ce2d00f464cbd3469655f4075e0810f2f18.idx
03/17/2021  08:28 AM         3,723,818 pack-91bb1ce2d00f464cbd3469655f4075e0810f2f18.pack
08/04/2020  02:46 PM            14,120 pack-933a0ff3e3e0a0eb02108ba0d8da7df949c82dae.idx
08/06/2020  09:43 AM           910,392 pack-933a0ff3e3e0a0eb02108ba0d8da7df949c82dae.pack
08/24/2020  08:27 AM            52,256 pack-9670f0b2e160e4a50c195363e7340e7c47db8bd2.idx
08/24/2020  08:27 AM         2,588,198 pack-9670f0b2e160e4a50c195363e7340e7c47db8bd2.pack
09/16/2020  10:42 AM            32,012 pack-97240f9262d0980ff06f44598d17a5913d2cfc06.idx
09/16/2020  10:42 AM         2,501,190 pack-97240f9262d0980ff06f44598d17a5913d2cfc06.pack
11/12/2020  10:26 AM            42,596 pack-985af2f2a323508abc732971a0dcd2bbad33515d.idx
11/12/2020  10:26 AM         3,305,479 pack-985af2f2a323508abc732971a0dcd2bbad33515d.pack
04/09/2021  07:59 AM            19,272 pack-9b01f79352c78cc1ad817da71c47986653f0ad78.idx
04/09/2021  07:59 AM         1,834,097 pack-9b01f79352c78cc1ad817da71c47986653f0ad78.pack
09/23/2020  02:20 PM            47,832 pack-9b9915c71fbc5048ca00ec7567bb076c22db475d.idx
09/23/2020  02:20 PM         3,634,444 pack-9b9915c71fbc5048ca00ec7567bb076c22db475d.pack
09/08/2020  08:31 AM            48,112 pack-9d22bbd40535618666243393e8ecbc87caf53451.idx
09/08/2020  08:31 AM         3,376,049 pack-9d22bbd40535618666243393e8ecbc87caf53451.pack
11/10/2020  03:10 PM            30,416 pack-9dc893a2377bbd973e41cb25dfb17be3a4ad3ba1.idx
11/10/2020  04:55 PM         2,270,019 pack-9dc893a2377bbd973e41cb25dfb17be3a4ad3ba1.pack
03/20/2021  09:09 AM            17,648 pack-9f5c1a840b8850e338de646888bc95eca8274cea.idx
03/22/2021  03:47 PM         2,249,700 pack-9f5c1a840b8850e338de646888bc95eca8274cea.pack
12/02/2020  08:17 AM            32,964 pack-a0770e99c3def71ac624a11ccddf177418b98808.idx
12/02/2020  08:17 AM         2,263,076 pack-a0770e99c3def71ac624a11ccddf177418b98808.pack
08/20/2020  07:42 AM            23,080 pack-a158cfc23069a5dff70b61028d1f687c146e9b50.idx
08/20/2020  07:42 AM         1,425,144 pack-a158cfc23069a5dff70b61028d1f687c146e9b50.pack
10/06/2020  07:16 AM            49,568 pack-a20e5472a2b233cfef31d95af853ed887e6ef26a.idx
10/06/2020  07:16 AM         5,221,047 pack-a20e5472a2b233cfef31d95af853ed887e6ef26a.pack
12/29/2020  09:37 AM            82,776 pack-a571e412546fe7d9bae11702f19fa68e85da7d18.idx
12/29/2020  09:37 AM         6,614,377 pack-a571e412546fe7d9bae11702f19fa68e85da7d18.pack
12/03/2020  01:42 PM            32,460 pack-ab85e3618aa2e107fa9c0b26ea4c1952032efa81.idx
12/03/2020  01:42 PM         1,890,888 pack-ab85e3618aa2e107fa9c0b26ea4c1952032efa81.pack
08/06/2020  09:37 AM            17,032 pack-b07a34ded940420dad43dc214f78297d8c79e103.idx
08/06/2020  09:52 AM         1,093,783 pack-b07a34ded940420dad43dc214f78297d8c79e103.pack
02/04/2021  12:52 PM            27,112 pack-b2ab9c8c33a206737dd0d2d327709551e628a043.idx
02/04/2021  12:52 PM         2,036,715 pack-b2ab9c8c33a206737dd0d2d327709551e628a043.pack
03/02/2021  07:49 AM            26,412 pack-b34328050a7094f42128ffdc79812e0d1a4782fb.idx
03/02/2021  07:49 AM         1,679,994 pack-b34328050a7094f42128ffdc79812e0d1a4782fb.pack
09/29/2020  10:10 AM             8,688 pack-b5d43c2d2b662936c84d34072e6248049be47e60.idx
09/29/2020  10:10 AM           884,969 pack-b5d43c2d2b662936c84d34072e6248049be47e60.pack
09/23/2020  03:30 PM             6,280 pack-b8bde221b1bec40696656009f50bf0239adc9ef7.idx
09/23/2020  03:30 PM           566,013 pack-b8bde221b1bec40696656009f50bf0239adc9ef7.pack
02/23/2021  11:24 AM            25,152 pack-b8f344ce951f82d568eaa59583527187fd10f999.idx
02/23/2021  11:24 AM         1,114,020 pack-b8f344ce951f82d568eaa59583527187fd10f999.pack
09/14/2020  11:40 AM            31,536 pack-b8f64a18dc81a02f797d20f3d5820e15b75bd4ad.idx
09/14/2020  11:40 AM         3,000,974 pack-b8f64a18dc81a02f797d20f3d5820e15b75bd4ad.pack
08/04/2020  08:42 AM             9,248 pack-ba3a186876fe4979a8f124c67e68afc97e07935d.idx
08/04/2020  08:42 AM         1,027,919 pack-ba3a186876fe4979a8f124c67e68afc97e07935d.pack
03/11/2021  12:12 PM            46,544 pack-bab71f246ac53517686fca7aa5001a695336de6d.idx
03/11/2021  12:44 PM         2,535,125 pack-bab71f246ac53517686fca7aa5001a695336de6d.pack
03/02/2021  10:43 AM             5,860 pack-bb54e4b947f771a64e5caebab8265af69d126bd0.idx
03/02/2021  10:43 AM           438,355 pack-bb54e4b947f771a64e5caebab8265af69d126bd0.pack
03/31/2021  10:02 AM            33,356 pack-bf8a81dd4bfd80e06ebb31c2376c8e7883e17e84.idx
03/31/2021  10:02 AM         2,302,249 pack-bf8a81dd4bfd80e06ebb31c2376c8e7883e17e84.pack
09/01/2020  06:53 AM            27,588 pack-bfb19de17acf70480f157d7238e3cc7d182f150a.idx
09/01/2020  06:53 AM         2,065,733 pack-bfb19de17acf70480f157d7238e3cc7d182f150a.pack
01/12/2021  09:10 AM            94,172 pack-c3c0d7bb4630ff422f9a36864eb53a63cc53cbd7.idx
01/12/2021  09:10 AM         4,912,214 pack-c3c0d7bb4630ff422f9a36864eb53a63cc53cbd7.pack
01/05/2021  08:47 AM            45,032 pack-c5a21a8b6d524177ee78b688650ae61e7ec4166e.idx
01/05/2021  08:47 AM         3,643,010 pack-c5a21a8b6d524177ee78b688650ae61e7ec4166e.pack
10/01/2020  11:53 AM            42,680 pack-c9406de06d0a0cc435c5e58bc8cb8cfd2549fd73.idx
10/01/2020  12:09 PM         2,661,417 pack-c9406de06d0a0cc435c5e58bc8cb8cfd2549fd73.pack
03/17/2021  01:49 PM            11,376 pack-c9cee6cf48d0d9c2a7ca1b9949ea0cec0284a266.idx
03/17/2021  01:49 PM         1,166,344 pack-c9cee6cf48d0d9c2a7ca1b9949ea0cec0284a266.pack
10/19/2020  07:39 AM            24,536 pack-cb61bfbb0abcf5e0dbdad5c76a9db4d305ef547a.idx
10/19/2020  07:39 AM         2,157,755 pack-cb61bfbb0abcf5e0dbdad5c76a9db4d305ef547a.pack
07/20/2020  10:20 AM        16,062,124 pack-ce3d47e72afdf1590f202e297e12ca5d8b68ec44.idx
03/17/2021  08:28 AM       248,384,478 pack-ce3d47e72afdf1590f202e297e12ca5d8b68ec44.pack
03/08/2021  04:00 PM            43,436 pack-ce60697498609309686c89907c7e998155b70a15.idx
03/09/2021  01:27 PM         3,849,985 pack-ce60697498609309686c89907c7e998155b70a15.pack
11/23/2020  10:39 AM            58,136 pack-d01b2869dd9a3e024d38d690999fa370b4e17693.idx
11/23/2020  11:56 AM         4,149,900 pack-d01b2869dd9a3e024d38d690999fa370b4e17693.pack
12/22/2020  09:59 AM            94,788 pack-d0bec3c16cca20b65265efe3b09143c1051feb24.idx
12/22/2020  09:59 AM         7,137,702 pack-d0bec3c16cca20b65265efe3b09143c1051feb24.pack
02/27/2021  09:45 AM            25,936 pack-d3ce40bedf9c60e7b9433f7ae06c64fdbeeadba7.idx
02/27/2021  12:57 PM         2,133,322 pack-d3ce40bedf9c60e7b9433f7ae06c64fdbeeadba7.pack
03/16/2021  12:18 PM             8,856 pack-d43f6a2eb7b53bde41c02c200c68b7e616e55f94.idx
03/16/2021  12:18 PM           730,887 pack-d43f6a2eb7b53bde41c02c200c68b7e616e55f94.pack
02/05/2021  12:17 PM            39,628 pack-d45be1f6024f503dc0c34f5a53d6d45507d5edb0.idx
02/05/2021  12:17 PM         3,518,972 pack-d45be1f6024f503dc0c34f5a53d6d45507d5edb0.pack
12/22/2020  03:24 PM            12,944 pack-d52783d990ed4fa0626cf62a0782023d181845f3.idx
12/22/2020  03:25 PM           659,511 pack-d52783d990ed4fa0626cf62a0782023d181845f3.pack
03/11/2021  07:01 PM            15,268 pack-d8d692c6daa2025d9b8078742b8067e1b355a09c.idx
03/11/2021  07:01 PM         1,511,879 pack-d8d692c6daa2025d9b8078742b8067e1b355a09c.pack
08/18/2020  08:43 AM            45,116 pack-da81ca8c46fc64b054fe571435df15502697064c.idx
08/18/2020  08:43 AM         3,712,332 pack-da81ca8c46fc64b054fe571435df15502697064c.pack
09/03/2020  08:45 AM            55,252 pack-dd23585a5d27bfec93827bfaeb2dcc48a739c99f.idx
09/03/2020  08:45 AM         3,952,032 pack-dd23585a5d27bfec93827bfaeb2dcc48a739c99f.pack
08/05/2020  12:48 PM            19,524 pack-decd70896c329b3a265cba079203cc337dca7c28.idx
08/06/2020  09:43 AM         1,766,307 pack-decd70896c329b3a265cba079203cc337dca7c28.pack
02/08/2021  12:39 PM            43,660 pack-dfdc8db6552813b23daf274338babd3b37693f2f.idx
02/08/2021  12:39 PM         3,039,969 pack-dfdc8db6552813b23daf274338babd3b37693f2f.pack
10/01/2020  01:40 PM            10,872 pack-e5e4725c7b8f3a29bcd81f0e18ff68c64ff33c62.idx
10/01/2020  01:40 PM           966,259 pack-e5e4725c7b8f3a29bcd81f0e18ff68c64ff33c62.pack
09/30/2020  07:37 AM            26,524 pack-e60e339a34ffa21487c8c593fbd500011a98b285.idx
10/01/2020  12:09 PM         1,771,974 pack-e60e339a34ffa21487c8c593fbd500011a98b285.pack
09/09/2020  08:56 AM            49,652 pack-e90c16f9ecd313235b5b4268a01cfd1806a48216.idx
09/09/2020  08:56 AM         3,213,624 pack-e90c16f9ecd313235b5b4268a01cfd1806a48216.pack
11/05/2020  10:35 AM            25,292 pack-e96e488f0be40d600cdb73ddda7ad3c38aad477c.idx
11/05/2020  10:35 AM         2,168,434 pack-e96e488f0be40d600cdb73ddda7ad3c38aad477c.pack
11/18/2020  01:27 PM            12,104 pack-eaddf47738cd4ead0e532c85a69ec99cb938c2a6.idx
11/18/2020  01:27 PM         1,360,602 pack-eaddf47738cd4ead0e532c85a69ec99cb938c2a6.pack
07/24/2020  04:49 PM            26,244 pack-eddf2fee745d93349babc0ab04545192a141c873.idx
07/24/2020  04:57 PM         1,977,277 pack-eddf2fee745d93349babc0ab04545192a141c873.pack
03/03/2021  11:58 AM            30,836 pack-ee3e3f0aee2cb605597e1a85b75b533a473d1316.idx
03/04/2021  08:46 PM         2,727,769 pack-ee3e3f0aee2cb605597e1a85b75b533a473d1316.pack
04/05/2021  08:07 AM            43,436 pack-eeaf040c76681eddac82bfafef27850dd3c913a2.idx
04/05/2021  08:07 AM         4,103,828 pack-eeaf040c76681eddac82bfafef27850dd3c913a2.pack
02/10/2021  09:43 AM            43,184 pack-f1e8072bfde37a422a70b0e58fd5990ff1900e54.idx
02/10/2021  09:43 AM         3,182,217 pack-f1e8072bfde37a422a70b0e58fd5990ff1900e54.pack
03/09/2021  12:50 PM            23,080 pack-f426c6ec21410346ae23d15a26a23a64db2cbdbd.idx
03/09/2021  01:27 PM         1,978,187 pack-f426c6ec21410346ae23d15a26a23a64db2cbdbd.pack
08/26/2020  02:56 PM            27,532 pack-f4b7d648ca91eddf2f601e631d3b831a941ecee5.idx
08/26/2020  02:56 PM         2,522,700 pack-f4b7d648ca91eddf2f601e631d3b831a941ecee5.pack
01/19/2021  09:11 AM            37,052 pack-f4f7bc0f18322f4c0196b9b806b84202c9fe7243.idx
01/19/2021  09:11 AM         3,581,798 pack-f4f7bc0f18322f4c0196b9b806b84202c9fe7243.pack
09/15/2020  10:36 AM            17,172 pack-fb748fe5a0238cff7a6c30b1dafc4580d4a19e10.idx
09/15/2020  10:36 AM         1,035,589 pack-fb748fe5a0238cff7a6c30b1dafc4580d4a19e10.pack
08/14/2020  08:56 AM            12,888 pack-fc9f855b62f2e2c5f77fffac02aecf1b4b32af03.idx
08/14/2020  10:45 AM         1,043,188 pack-fc9f855b62f2e2c5f77fffac02aecf1b4b32af03.pack
             282 File(s)    756,021,106 bytes
               2 Dir(s)  89,487,589,376 bytes free

@AArnott
Copy link
Collaborator

AArnott commented Apr 10, 2021

None of these files look very large, but I suppose perhaps the 129MB+ files might be hard to map into contiguous memory that large.
@qmfrederik I believe when memory mapping files that it's possible to map only portions of files at a time. I wonder if a fix for this would be to set a max "window" size to map into memory and move it as necessary?

@qmfrederik qmfrederik mentioned this issue Apr 27, 2021
Closed
@filipnavara
Copy link
Member

Here's one more data point:

filipnavara@172-4-1-17 emclient % ls -l .git/objects/pack
total 8302608
-r--r--r--  1 filipnavara  staff    14635208 Jun 11 10:00 pack-80c7ee391de74c1da6faf79c8b49c68d8c2dacf6.idx
-r--r--r--  1 filipnavara  staff  4236296121 Jun 11 10:00 pack-80c7ee391de74c1da6faf79c8b49c68d8c2dacf6.pack

Mapping the index files is fine but mapping the whole pack files is definitely problematic.

@filipnavara
Copy link
Member

filipnavara commented Jun 14, 2021 •
edited
Loading

Here's an experiment with mapping only small chunks of the pack file: https://github.com/dotnet/Nerdbank.GitVersioning/compare/master...filipnavara:big-pack?expand=1

The approach is not perfect but it highlights few things:

  • There's unused code for GitPackPooledStream.
  • It's likely going to have performance impact so it may be worth the trouble to choose different strategy for small pack files, big pack files and/or 32-bit/64-bit processes.
  • Not sure what is the motivation for MemoryMappedStream compared to the built-in MemoryMappedViewStream. It could be performance or different behaviour when disposing. Either way it doesn't handle non-aligned offsets. The memory mappings have to be aligned to page size boundaries on the OS level APIs. .NET tries to hide this restriction by aligning the mapped block to nearest smaller offset and then making the window large enough. Then it makes the stream point to the actual offset within the first page of the view.
  • Mapping small chunks may turn out to be less efficient in worst case. The mapping is always aligned to OS page size so if we create a view for each individual object then two small adjacent objects will end up with two bigger views. Moreover, to read the object size we have to first map a bit of the file to read the object header. I optimistically start with an 8Kb view, read the header and if the object is bigger, then remap to a bigger view. This is somewhat suboptimal.

In the end I have a feeling that it may be more reasonable to map only the index files and not the pack files. If it's a problem performance wise it's probably fine to do that only as a fallback when certain parameters are exceeded (file size threshold, 32-bit process, the mapping fails, etc.). .NET 6 will have better APIs to handle this kind of random reads (dotnet/runtime#53669) but unfortunately that's still some months away.

@AArnott
Copy link
Collaborator

AArnott commented Jun 14, 2021

@qmfrederik Do you have time to investigate @filipnavara's data and proposal?

@filipnavara
Copy link
Member

I'd be happy to do some more testing and try out other approaches or even run benchmarks. However, with the window for .NET 6 runtime changes slowly closing in I need to focus on completing some work there before moving on to other projects.

@qmfrederik
Copy link
Contributor

All help is welcome, so I'd be more than happy if @filipnavara wants to invest some time in this 😄 .

Not sure what is the motivation for MemoryMappedStream compared to the built-in MemoryMappedViewStream

I think the main reason was .Dispose behavior. Pooling streams had a significant impact on performance. There was an attempt to use the built-in version (0c5a573) which was reverted (61e877a, #521 (comment)).

In the end I have a feeling that it may be more reasonable to map only the index files and not the pack files.

Yes, that may be the case. If I remember correctly, the stream pooling and usage of memory mapped files was introduced at about the same time.
There's probably a real benefit for index files, since we do a binary search there.

It is possible that the memory mapping doesn't have much of a perf impact on pack files, and the real gain comes from stream pooling.

There are a couple of perf tests which run as part of CI, so it should be fairly easy to tell whether changes regressed performance or not 😃

@AArnott
Copy link
Collaborator

AArnott commented Jun 21, 2021

Keep in mind the workaround we have of setting the NBGV_GitEngine environment variable to LibGit2 so you don't have to stick with the 3.3 version, folks.

@AArnott AArnott mentioned this issue Jun 28, 2021
Merged
@AArnott AArnott changed the title IOException when building IOException when building from mapping pack files into memory Jun 28, 2021
@AArnott AArnott mentioned this issue Jul 9, 2021
Closed
@benmccallum
Copy link

benmccallum commented Jul 23, 2021 •
edited
Loading

I'm now starting to run into this in local dev too in VS. I guess VS 2022 can't come fast enough :P

Searching for a way to configure the use of 64-bit msbuild from VS but that doesn't seem to be possible. Guessing that's a limitation of a 32-bit process not being able to spawn a 64-bit one, but this certainly isn't my area of expertise.
https://docs.microsoft.com/en-us/visualstudio/msbuild/msbuild?view=vs-2019

So rather than need to run msbuild from cmdline, is there a way to perhaps toggle off ngvb in local dev? Which would probably be something we'd want in any case to avoid the compile-time overhead...

Is toggling it out of the csproj if not Release configuration a viable alternative? Can't say I've ever toggled off a package ref by Configuration. Feels a bit wrong; but trying to avoid getting the whole team to set an env var.

<PackageReference Include="Nerdbank.GitVersioning" Condition=" '$(Configuration)' == 'Release' ">
  <Version>3.4.190</Version>
  <PrivateAssets>all</PrivateAssets>
</PackageReference>

@qmfrederik
Copy link
Contributor

@benmccallum If you have the time to dig into this, could you give #626 a try and let us know if this fixes the problem for you? There may be a perf regression n macOS, but since you mention VS, I guess you're on Windows so you should be good.

Alternatively, set the NBGV_GitEngine environment variable to LibGit2 to sidestep the managed implementation altogether.

@AArnott
Copy link
Collaborator

AArnott commented Jul 23, 2021

Guessing that's a limitation of a 32-bit process not being able to spawn a 64-bit one, but this certainly isn't my area of expertise.

There's never a problem with 32-bit processes spawning 64-bit ones. It's just that that's not what msbuild in VS was designed to do.

@benmccallum
Copy link

@qmfrederik, I can look to give #626 a go at some point, is it available as a nuget preview release somewhere?

@AArnott
Copy link
Collaborator

AArnott commented Jul 23, 2021

@benmccallum It used to be, but it got GC'd by Azure Pipelines. I've queued a new build here: https://dev.azure.com/andrewarnott/OSS/_build/results?buildId=4966&view=results
You should be able to download the nupkg from the pipeline artifacts when it's done.

@AArnott AArnott closed this as completed Jul 27, 2021
@AArnott
Copy link
Collaborator

AArnott commented Jul 27, 2021

This is now available in https://github.com/dotnet/Nerdbank.GitVersioning/releases/tag/v3.4.228

@benmccallum
Copy link

Thanks @AArnott , sorry I didn't get around to give feedback in time. Will get on the new version tomorrow and give it a whirl.

@benmccallum
Copy link

Seems to be working well for us on all platforms, 👍🏻 Thanks again

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug managed-git
Projects
None yet
Milestone
v3.4
Development

Successfully merging a pull request may close this issue.

GitPack: Don't use memory mapped streams when accessing pack files
5 participants
@AArnott @benmccallum @filipnavara @ssa3512 @qmfrederik