-
Notifications
You must be signed in to change notification settings - Fork 88
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Zip64 write support #230
base: master
Are you sure you want to change the base?
Conversation
I've read the code some more and I don't see that fflate adds an |
I did a little research and experimented with other zip programs, mainly winrar, 7z, native windows zip app and linux zip command, and I found out that:
I have added a new special flag which when set results in the addition of an empty Zip64 EF and consequently a DD in Zip64 format to the end of the file stream. Here is the code to test this in practice: const entries = [
[ 'big file', new Uint8Array(512 * 1024 * 1024).fill(10), 10 ], // ~5.4GB of data
[ 'not quite as big', new Uint8Array(512 * 1024 * 1024).fill(11), 5 ], // ~2.7GB of data
[ 'pretty average', fflate.strToU8('file content') ],
]
const writeStream = createWriteStream('./zip64-test.zip');
const zip = new fflate.Zip((err, chunk, final) => {
if (err) throw err;
if (final) writeStream.end(chunk);
else writeStream.write(chunk);
});
for (const [ fileName, content, rep = 1 ] of entries) {
const file = new fflate.ZipDeflate(fileName, { level: 8 });
file.zip64 = content.length * rep > 0xffffffff; // must be set 'manually' as stream is usually of unknown length
file.mtime = Date.parse('2000-01-01'); // for consistency
zip.add(file);
for (let i = 1; i <= rep; i++) {
file.push(content, i === rep);
}
}
zip.end(); On my machine it takes a little over a minute to generate this file.
|
Thank you for the PR! Overall looks mostly good, bar some stylistic decisions, but I'll do a more thorough review when I get a chance. Unfortunately life has kept me quite busy of late so I'm not sure when that will be :/ I think overall it would be a good idea to properly support Zip64 writes, but I'd like to verify that this doesn't cost too much in terms of bundle size. If it's a huge change in bundle size it may not be worth it as this is a relatively niche use case for this library. |
Sure no pressure, this PR can wait here 👍
Which I personally find quite acceptable value in exchange for increased tool reliability and the ability to create zip files up to 9 Petabytes! Edit:
and now it can be said that this PR has reached net-zero emissions of bundle size. And those saved 163 bytes could even be sold to other features as bundle-size credits! |
Resolves #229
Add support for automatic generation of
Zip64 end of central directory record
andZip64 end of central directory locator
when bytes available in standardEnd of central directory record
are not enough.Zip64 EOCD will be added when when at least one of the following is true:
1. total number of files in .zip archive is greater than 65,535
2. size of the central directory is greater than 4,294,967,295 bytes (4GB)
3. central directory offset is greater than 4,294,967,295 bytes (4GB)
Add support for automatic generation of
Zip64 Extended Information Extra Field
when bytes available in standardlocal file header
orcentral directory file header
are not enough.Zip64 Extra Field with 2 fields will be added to
local file header
when:1. file uncompressed/original size is greater than 4,294,967,295 bytes (4GB) or
2. file compressed size is greater than 4,294,967,295 bytes (4GB)
Zip64 Extra Field with 1,2 or 3 fields will be added to
central directory file header
when:1. file uncompressed/original size is greater than 4,294,967,295 bytes (4GB) [1 field or more]
2. file compressed size is greater than 4,294,967,295 bytes (4GB) [2 fields or more]
3. local file header offset is greater than 4,294,967,295 bytes (4GB) [3 fields]
Add support for
Data descriptors
in Zip64 format activated by a new special flagzip64
.For file streams with this flag, an empty Zip64 Extra Field with 0 fields will be added to
local file header
and then, at the end of the file stream, a Data Descriptor in Zip64 format.Other changes:
mtime
generated during zip stream. Currently, if the time between adding a file to the archivezip.add(myFile)
and ending the streamzip.end()
is greater than a few seconds, themtime
in the Local File Header differs from themtime
of the Central Directory Header. Assuming that the user has not definedmyFile.mtime = some date
.zip.add(myFile), myFile.push(strToU8('the end'), true)
. Currently, during the zip stream, the Data Descriptor is added to every file in the archive even though this is not always necessary, and as a result it increases the final size of the zip file as well as making it more difficult to read from the stream later.zipSync
andzip (async)
functions. Regarding the creation of the zip file header.