Skip to content

NuGet: read .nuspec manifests in UTF-16 with a byte-order mark - #865

Merged
schmitch merged 1 commit into
mainfrom
claude/597-nuspec-utf16
Nov 3, 2026
Merged

schmitch merged 1 commit into
mainfrom
claude/597-nuspec-utf16

Conversation

@schmitch

@schmitch schmitch commented Nov 3, 2026

Copy link
Copy Markdown
Member

Fixes #597

What

parse_nupkg refused a manifest in UTF-16 LE with a byte-order mark (Microsoft.Web.Infrastructure 1.0.0 on nuget.org) with cannot decode input using UTF-8. Because of that, the package could not be pushed, and a proxy of nuget.org could not serve it.

parse_nuspec (crates/formats/nuget/src/nuspec.rs) now runs every manifest through a new decode_manifest first. It reads the manifest the way NuGet's XDocument.Load does:

  • a UTF-16 LE (FF FE) or BE (FE FF) byte-order mark: the rest is decoded to UTF-8. An odd byte count or an unpaired surrogate is refused as a malformed manifest.
  • a UTF-8 mark (EF BB BF): the mark is dropped.
  • no mark: the bytes are read as UTF-8, as before.

Every other check (DOCTYPE, depth, identity, ids, versions, limits) then runs on the decoded text. The size limits still apply to the raw bytes. All callers go through parse_nuspec: push, proxy manifest verification (manifest_identity), registration dependency groups and display metadata. The stored manifest keeps the package's own bytes, so it is served unchanged.

Tests

  • New fixture fixtures/nuget/utf16.package.1.0.0.nupkg, generated by fixtures/nuget/make_fixtures.py. Its manifest is UTF-16 LE with a BOM, encoding="utf-16" and non-ASCII text. It is the only fixture added; the existing fixtures are left unchanged.
  • tests/parse.rs::parses_a_utf16_manifest_with_a_byte_order_mark parses the fixture end to end and reads the metadata back from the stored manifest.
  • Unit tests in nuspec.rs:
    • manifests_with_a_byte_order_mark_are_decoded_as_nuget_does covers UTF-16 LE, UTF-16 BE and UTF-8 with a BOM.
    • malformed_utf16_manifests_are_refused covers an odd byte count and an unpaired surrogate, and checks that the DOCTYPE refusal still applies after decoding.
  • With the decoding disabled, both unit tests fail.
  • Run before pushing: cargo fmt --all --check, cargo clippy -p forge-format-nuget --all-targets -- -D warnings, cargo test -p forge-format-nuget and cargo xtask changelog --check.

Changelog

docs/changes/2.2.0/597.fixed.md


Generated by Claude Code

parse_nupkg refused a manifest in UTF-16 LE with a byte-order mark
(Microsoft.Web.Infrastructure 1.0.0 on nuget.org), so it could neither be
pushed nor served through a proxy of nuget.org. The manifest is now
decoded the way NuGet's XDocument.Load reads it: a UTF-16 LE or BE mark is
decoded, a UTF-8 mark is dropped, and every other check still runs on the
decoded text. The stored manifest keeps the package's own bytes.

Adds a UTF-16 manifest fixture (fixtures/nuget/make_fixtures.py).

Fixes #597

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012e6DJaoNeeYjbTdvTzQ728
@schmitch
schmitch enabled auto-merge November 3, 2026 21:28

schmitch commented Nov 3, 2026

Copy link
Copy Markdown
Member Author

CI test failed on de68c0f in one test, dek_revocation_is_checked_at_commit_time_against_writers (crates/db/tests/encryption.rs:607, unwrap_err() on Ok). Every other test binary passed, including all of forge-format-nuget.

I don't think this failure comes from this PR:

  • This PR only changes NuGet manifest parsing (crates/formats/nuget), a fixture and a changelog fragment. It doesn't touch crates/db, the secret DAO or the encryption keys.
  • The test checks that DEK revocation is seen at commit time by a write blocked on a PostgreSQL row lock. It depends on timing: a fixed 500 ms sleep, then the revocation commits while the write waits.

I've re-run the failed job once to confirm. There is no fix for that test to port into this PR. If it fails again, I'll report it as a separate issue rather than widen this PR.


Generated by Claude Code

@schmitch
schmitch added this pull request to the merge queue Nov 3, 2026
Merged via the queue into main with commit e186e6a Nov 3, 2026
33 of 34 checks passed
@schmitch
schmitch deleted the claude/597-nuspec-utf16 branch November 3, 2026 22:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

NuGet: a package whose .nuspec is UTF-16 (Microsoft.Web.Infrastructure 1.0.0) is refused, also through a proxy of nuget.org

2 participants