Error: invalid central directory file header signature when unzipping big files, zipped by mac os

[d-mg]

Using the terminal to zip them there is no problems with unzipping. But when I use the GUI for zipping, and only with bigger files (around 8 GB and up) I get the following error:

Error: invalid central directory file header signature: 0x2d8c9fa4
at /Users/dmg_/projects/test-zipper/node_modules/yauzl/index.js:253:70
at /Users/dmg_/projects/test-zipper/node_modules/yauzl/index.js:622:5
at /Users/dmg_/projects/test-zipper/node_modules/fd-slicer/index.js:32:7
at FSReqWrap.wrapper [as oncomplete] (fs.js:658:17)

Also when running zipinfo I get:

Archive: Archive.zip
Zip file size: 5946143505 bytes, number of entries: 140
warning [Archive.zip]: 4294967296 extra bytes at beginning or within zipfile
(attempting to process anyway)

I did some digging through issues and some searching and I'm pretty sure these extra bytes make it that Yauzl can't find the header of the central directory.

Do you have any clue why GUI Mac OS zipped zipfiles have these extra bytes? I do see it puts some extra '__MACOSX' folders in the zip and suspect the presence of these to have something to do with it, but that's more of a guess.

[thejoshwolfe]

Thanks for the report. Is it possible to provide a download for a zip file that reproduces this problem? You mention it happens with very large files, like 8GB, so perhaps hosting a download would be infeasible.

Alternatively, could you provide minimal steps to reproduce the issue? Perhaps something like:

1. head -c 8000000000 /dev/urandom > file.bin
2. right click on file.bin, and choose something.

I don't have a Mac, so i'd be forwarding these instructions to a friend who can transfer me an 8GB file easily.

[thejoshwolfe]

Do you have any clue why GUI Mac OS zipped zipfiles ...

An easy answer is: yes, I have a clue. The Mac Archive Utility is a piece of trash when it comes to the zip file format. This will be the fourth (I believe) compatibility issue that boils down to Apple getting the zip file format embarrassingly wrong.

I'm not surprised at all that this is an issue. In fact, I previously thought that Archive Utility was simply unable to handle zip files larger than 4GB due to lack of ZIP64 support. Are you sure the zip files that you create are readable by anyone? Like, can the GUI itself read its own files that it creates that are 8GB in size? If so, then there's some strange format at play that isn't ZIP64 format.

[d-mg]

Did some more testing and it seems to fail somewhere between 4.19 and 4.33 GB.

Since I'm trying to zip 4k mp4's I used ffmpeg to create files (warning, don't watch them with your volume opened completely) https://www.ffmpeg.org/download.html

this creates a 32 seconds mp4 (4.19 GB)
ffmpeg -f rawvideo -video_size 3840x2160 -pixel_format yuv420p -framerate 25 -i /dev/urandom -ar 48000 -ac 2 -f s16le -i /dev/urandom -c:a aac -t 32 output32secs.mp4

this creates a 33 seconds mp4 (4.33 GB)
ffmpeg -f rawvideo -video_size 3840x2160 -pixel_format yuv420p -framerate 25 -i /dev/urandom -ar 48000 -ac 2 -f s16le -i /dev/urandom -c:a aac -t 33 output33secs.mp4

The 32 seconds one unzips fine with yauzl, the 33 seconds (or any larger) one fails with:
Error: invalid central directory file header signature: XXXXXXXX.

I zip them by simply going into Finder on my Mac, right clicking the file and choosing compress "output33secs.mp4".
The resulting zip file causes the error when unzipping with Yauzl.
Zipping them through the command line zip somezip output33secs.mp4 creates a zip file that Yauzl unzips just fine.

Hope this is enough to reproduce the problem, if not let me know.

[thejoshwolfe]

Thanks, I've reproduced the issue. Indeed, the Info-ZIP unzip command line program can't read this file correctly either, so this issue is not specific to yauzl.

I'll do some research and tell report back here. (It may take a few weeks.) I'm curious to see what the hell Archive Utility did that it can read its own malformed zipfile but no one else can.

[thejoshwolfe]

I've determined what the issue is. It's pretty much the same problem as #60.

When the Mac Archive Utility needs to write 32-bit values in the metadata, it doesn't check for overflow and simply writes the lower 32 bits of the value. There are 3 values that get corrupted like this in the case of archiving a file that's over 4 GB in size: the "offset of start of central directory with respect to the starting disk number" in the end of central directory record and the "compressed size" and "uncompressed size" fields of the local file header and central directory file header (and data descriptor, but that doesn't matter too much).

This means that the metadata is an outright lie. This is absolutely a bug in Archive Utility, and it's a nasty one too. The correct behavior would have been to use ZIP64 format, but Archive Utility has no idea what that is.

There may or may not be any way to correctly and reliably recover from this corruption. I'm still thinking about it.

[d-mg]

Thank you for looking into it. Also I noticed that sometimes the unzip CLI tries and unzips part of the file but they you lose half of the video file and it's completely not reliable. Hope you do find a solution, but in the meantime my workaround is just avoiding the Archive Utility. I did send a report to apple, if they get back to me with any relevant information I will share that here as well.

[thejoshwolfe]

Oh. How did you do that? I was unable to find any bug tracker for either Archive Utility or the ditto command line program, which is allegedly the backend for Archive Utility's zipfile functionality.

If Archive Utility's zipfile implementation were open source, i could even make a pull request fixing their bugs, but I didn't find that either.

[d-mg]

I reported it here: https://www.apple.com/feedback/macos.html
It's pretty much just sending feedback for macOs, they state that they will look at everything sent in there but that they might not get back to you. It's a long shot but who knows if people start nagging they might fix it...

[overlookmotel]

@d-mg Out of interest, what version of Mac OS did you create the faulty zip file on?

Absolutely bonkers that Mac OS has such a broken ZIP implementation, especially as it's meant to cater for "creatives" who are the most likely people to be zipping up large (e.g. video) files.

@thejoshwolfe Are you saying that Mac OS Archive Utility does not make use of ZIP64 extra fields at all?

[overlookmotel]

If the number of files in the ZIP is not enormous, I guess that "offset of start of central directory with respect to the starting disk number" can be deduced from the total length of the ZIP file and lower 32 bits of the offset recorded by MacOS.

Each central directory file header is max 196654 bytes (46 + 65536 + 65536 + 65536). So if the ZIP contains less than 21000 files (approx), the central directory cannot be bigger than 4GiB. So then the low 32 bits are enough to determine the location of start of the central directory.

That doesn't solve the problem of "compressed size" and "uncompressed size" (and also, presumably, the "relative offset of local file header") fields also being truncated to 32 bit. But I guess you could use the info that is available to step through the file in 4 GiB jumps to check all possible locations for matching local file headers.

Ergh! But do you think this would work?

My work involves receiving a lot of large ZIP files from sources all over the place, and I sadly have no way to prevent them being created with Mac OS Archive Utility (I suspect most are), so I would be motivated to help solve this problem.

I also have a massive collection of large ZIP files from varied sources to test an implementation on.

[d-mg]

@overlookmotel I was using High Sierra for this

[thejoshwolfe]

Ergh! But do you think this would work?

I developed a plan with a lot of the same ideas you presented to make this work, but it's very difficult to implement. The gist is to track a non-deterministic multistate of possibilities depending on what kind of corruption the zipfile has.

The user would need to explicitly request this complexity for three reasons: I don't want everyone to pay the performance cost of Apple's incompetence/hostility if they don't want to; the user needs to use the library differently when this complexity is enabled; the complexity can change the correct interpretation of a correct zipfile to be incorrect in rare cases.

The Mac Archive Utility behaves predictably, albeit incorrectly, when laying out data in the zipfile, so I can check if the zipfile is deviating from the Archive Utility-style. If we know we aren't dealing with Archive Utility, we can drop the complexity and behave normally. However, we can't be certain that we are dealing with Archive Utility until we verify that one of these overflow corruptions has happened.

Each case of corruption can be detected heuristically by seeking to the locations that would cause the overflow to give us the value we see, and checking the zipfile data structure signatures. This is not a perfect solution, because there may be a valid zipfile that perfectly mimics the Archive Utility corruption. That's the main problem with this Archive Utility bug is that it creates "valid" zipfiles with the wrong content. So whether we should assume that it's corrupted or correct is a guess, and that's one of the reasons why the user would have to explicitly ask for this feature to be enabled.

The probability of this "false positive" detection happening randomly is low, so it's not the worst thing about this idea.

The worst part is that this it overall sucks. It's really complicated, it's going to negatively impact performance at least a little bit for everyone, and it's all due to flagrant incompetence/hostility at Apple being widely adopted. My motivation is really the limiting factor for implementing a fix for this, but hearing people say that this is important to them really helps. 👍

@thejoshwolfe Are you saying that Mac OS Archive Utility does not make use of ZIP64 extra fields at all?

Yes. I haven't seen the source code, because I believe it's closed source, but all of my experience and experimentation with it suggest that the implementation is oblivious to the ZIP64 format entirely, and apparently also oblivious to the concept of integer overflow and bounds checking.

[overlookmotel]

@thejoshwolfe Thanks for your lengthy reply.

I agree that it totally sucks! But, at least for me, it would be really useful to resolve.

Complexity: To my mind, this is the biggest downside. It would make the code considerably more complex. I imagine it would likely require a refactor to split different steps of file parsing into separate functions, so functions can call themselves recursively as they try different possibilities. That's a lot of code churn.

Performance: As far as I can see, I doubt that it would impact performance in any significant way for non-Apple ZIP files. The workarounds for the Apple case, I think, only need kick in if a valid Central Directory Record cannot be found in the location it should be. So the execution path for non-Apple archives could remain the same.

False positives: I hope possibility of false positives could be reduced to zero, or at least to an infinitesimal level. I hope (but don't know yet) that Archive Utility ZIP files can be "fingerprinted" in some way. For example, maybe they always set the "version made by" field to a particular value which no-one else uses. If so, then the wacky heuristics could be disabled if it's not an Archive Utility file - so anything not made by Archive Utility would behave as before, and the only files which would be parsed differently would be ones which at present yauzl can't deal with anyway.

Explicit enabling: I agree this should have to be explicitly enabled by the user.

I would like to do some digging on this and see what I can find.

In principle, if (1) I can come up with something half-sensible which doesn't affect performance significantly or risk screwing up valid ZIP files and (2) am willing to do a lot of the work on implementation, would you likely be willing to accept the downside of additional code complexity in order to support Archive Utility files? (I totally understand if you don't)