Repository navigation
Conversation
Read the Packages indexes of the suites, components and architectures given as query parameters on the archive URL. Each index is checked against the SHA256 its suite's InRelease or Release lists, tried as Packages.xz, Packages.gz and Packages past a 404, and decoded as a stream. The LZMA2 dictionary of every .xz block is checked before decoding, since the decoder allocates whatever a block asks for. Artifacts record arch and distro (the suite's codename without -updates or -security) for qualifier selection. Versions keep their epoch and compare with the deb scheme. Add Client.Stream for large bodies that aren't JSON.
repository_url held only the archive root, so a PURL the registry built could not be read back: the suites and architectures were missing. It is now the whole configured URL. Keep the resolver test, which needs httptest.NewTLSServer, out of the TinyGo build.
Read repodata/repomd.xml, follow its primary entry, check that file against the checksum and size listed there, and decode .zst, .xz, .gz or plain XML as a stream. A zstd frame may ask for a window of at most 128 MiB, and .xz blocks are checked as in the Debian client, whose check moves to internal/xzcheck for both. Artifacts record arch, epoch and the distro given as a query parameter on the repository URL, since repository metadata names none. The version number is version-release, as in the rpm PURL type; versions are ordered by epoch, version and release.
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
|
The dictionary check found blocks through each stream's index, which the decoder only compares with the blocks after it has read their headers and allocated their dictionaries. An index that left a block out hid it. The check now reads the file front to back: each block header, then the block's LZMA2 chunks by the sizes in their headers. It no longer takes sizes from the footer, which could overflow a 32-bit int. - The same version in two releases is recorded for each, so either distro resolves; -updates repeating a version still adds nothing. - An index the Release lists but the archive doesn't serve fails the read instead of leaving its packages out for the Registry's lifetime. - A clearsigned Release with CRLF line endings is read, and one without its header separator or signature is an error, not an empty Release. - Dependencies keep << and >> requirements, which were cut as build profiles.
The forward-walking xz check replaces the one in internal/xzcheck, with its tests.
Checking block headers ahead of decoding cannot be made sound: where a block's data ends is only known by decoding it, since the LZMA2 decoder may stop short of the compressed size a chunk declares. A file could place a block header where the check did not look. internal/xzlimit reads the .xz container itself and hands each block's data to ulikunitz/xz's LZMA2 decoder with exactly the dictionary the block asks for, refusing one over the limit as it is reached. It verifies the header, block, index and footer checksums. Packages.xz is now decoded as a stream like the other variants, so its size limit and the check before decoding are gone.
primary.xml.xz is decoded through internal/xzlimit as a stream; the check ahead of decoding and internal/xzcheck are gone.
A block header may declare the compressed and uncompressed size of its data. xzlimit read past them; a block whose data has another size is now refused.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #108.
Builds on #140 (the Debian client), which adds
Client.Streamand the xz dependency. Until that merges, this PR shows its commits too; the RPM change is the last commit.Adds an
rpmregistry that reads a repository'srepodata.Configuration. The URL is one repository, the directory that holds
repodata/. Nothing is inferred from a mirror root, and mirror lists and metalinks are not followed. Repository metadata doesn't name its distribution, so thedistroto record on artifacts is a query parameter, as with the Debian client's suites:Metadata.
repodata/repomd.xmlis read and itsprimaryentry followed..zst,.xz,.gzand plain XML are decoded and parsed as a stream. Other formats, such as zchunk, fail before anything is downloaded..xzgoes throughinternal/xzlimitfrom Add a Debian repository client #140, which holds every block to a 64 MiB dictionary as it decodes.Versions and artifacts.
version-release, as in therpmPURL type, where the epoch is a qualifier. Versions are ordered by epoch, version and release with therpmscheme.sha256-..., orsha512-...on openSUSE), its size, and the metadataarch,epochanddistro.hrefor throughxml:base, gets no artifact.requiresentries that name packages. File paths, sonames and capabilities such asrpmlib(...)are left out, since resolving them needs every package'sprovides.Left for follow-ups:
epochqualifier in the resolver. Two builds that differ only in epoch are rare, and today the first one wins.repomd.xml.asc.primary_zck) and the sqliteprimary_db.Tested. Unit tests cover each compression, epoch and release ordering, architectures and
noarch, checksums and sizes, a missing URL, metadata larger than the buffered response cap, and not-found packages and versions. All of it is mutation-checked. I also ran it against real repositories:I downloaded one resolved
.rpmand checked it against its checksum.