python-stdlib/zipfile: Add read-only zipfile module.

[agatti]

Summary

This PR adds a limited implementation for the zipfile module, allowing enumerating Zip archives' contents and read data contained in it.

Only the parts of the zipfile module (and its classes) API that deal with contents enumeration and data reading are implemented (with optional decompression). Only files inside Zip archives using STORED and DEFLATED methods can be read, and the latter only if the deflate module is available.

What is not supported: encryption, archive creation, any other compression besides no-compression and standard deflate, large file support (ie. zip64), data descriptors, central directory headers, file extraction, crc32 checks, comments (both archive-wide and file-specific), the zipfile.Path class, thezipfile.PyZipFile class, the module's command line interface, and probably something else that's not really important.

In detail:

I guess there's nothing to elaborate w.r.t. lack of BZIP2/LZMA/ZSTD compression support or no encryption support :)

I did not include the constants for ZIP_LZMA and ZIP_ZSTANDARD, as they are not supported anyway. I can add them for compatibility reasons, but I doubt they're useful being there.

Calling zipfile.ZipInfo.setpassword or either zipfile.ZipInfo.open or zipfile.ZipInfo.read with the pwd argument not being None will raise NotImplementedError.

zipfile.ZipFile.extract and zipfile.ZipFile.extractall raise NotImplementedError. zipfile.ZipFile.extract can be written using zipfile.ZipFile.getinfo together with zipfile.ZipFile.open or zipfile.ZipFile.read if it is known there's enough memory. Same for zipfile.ZipFile.extractall, but using zipfile.ZipFile.infolist to iterate through the archive contents. This is usually best to implement on a case by case basis anyway (so buffer sizes can be tuned, write retries can be implemented, utf8 file names, and maybe yielding to the scheduler after each block).

zipfile.ZipFile.testzip raises NotImplementedError, for the same reasons as zipfile.ZipFile.extract and zipfile.ZipFile.extractall. CRCs only cover file data, not their metadata anyway, and that can be checked manually using the same iteration mechanism described in the previous section.

Data descriptors and Central directory headers require a bit more technical explanation (for the full description please read the official file format specification.

When creating archives, there may be situations in which the compressed/uncompressed file sizes and CRC32 are not known in advance, so what happens is that a header with null fields is prepended to the compressed data, a bit is set in the flags indicating this condition, the compressed data is written to the archive, and then the now known information is written after the compressed block. For simplicity reasons this is not supported, as unless I'm mistaken knowing when the block ends in a multi-file archive can only be done by dealing with the compressed data block at the bit-stream level. If a file with this flag bit set is found, enumeration bails out altogether.

Regarding central directory headers, the file format metadata grew by accretion, basically. So there are several layers of metadata in different positions, each with an ever-growing number of fields being contained. Right now what this code handle is the earliest structure type, with just name, modification timestamp, compression method, file sizes, and not much else being useful.

Things like file comments, original file permissions (remember this thing initially came out back when dinosaurs were still out there and MS-DOS was still a thing), UID, GID, and so on needed to be put into archives but the file header format couldn't be changed easily, so PKWARE added those records at the end of the file (hence being called central, of course.) This is something that may be supported if needed, but I don't think it's terribly useful for embedded applications anyway.

Same for Zip64, not sure about the usefulness of adding support for that - if it's ever possible, that is (internal filesystem pointers should not be 32-bits wide themselves).

Writing archives is possible in theory, but I'd rather have this being able to read archives first and then add write support later once most issues are sorted out (plus for embedded applications it's most probable you want to read archives' data anyway).

Same for the command line interface (ie. -m zipfile), right now it's not terribly useful and will add a fair bit of code to make it work on par with CPython.

Testing

I've added a test file with (I believe) enough code to more or less show this can at least handle files built without too many exotic options. However, the more archives (especially ones built on "modern" windows archivers) could help ironing out issues that may arise.

Running the micropython-lib test suite with the new test file in the files list still passes (and runs the new tests too!).

Trade-offs and Alternatives

Despite being a relatively large module, the footprint is 2906 bytes last time I checked. It can be brought down for sure, but most of the overhead is made up of method/class name strings, unfortunately.

Right now all file names must be encoded as ASCII, or file enumeration/open will fail. There's a bit in the flags set read by this module that indicates whether a file name has to be encoded as utf-8, but for space reasons that is not supported.

To let zipfile.ZipFile.open work as CPython I've had to write a stream proxy that exposes a virtual file offset and file size limit, using the file handle opened by the parent zipfile.ZipFile instance for read operations. This has the potential to fail if an archive file is read and the parent archive files are enumerated at the same time as the stream position may end up being out of sync, so multithread safety is not guaranteed.

zipfile.ZipInfo.is_dir assumes that every archive was built on Unix, so to find out if a record is a directory or not it looks for a file that's 0 bytes long and whose file name ends with a /. This is done for size reasons as the system type that created the archive is encoded in the header, so in theory I can look for a trailing \ if the archive was built under MS-DOS or Windows. However I don't know if modern Windows systems' zip archivers still use \ in path names or not.

zipfile.ZipFile.namelist could be made to raise NotImplementedError as it can be written by filtering the output of zipfile.ZipFile.infolist, but that's probably a bit too much.

Given how the spec was implemented, there may be files that will fail to open due to some quirk or some version-specific data that is not taken care of. I believe having this module in the current state is still better than nothing, as those issues can be sorted out in time once reported.

Generative AI

I did not use generative AI tools when creating this PR.

---

And this PR description has almost as many bytes as the module itself! Sorry for that.