Repository navigation
unpackb() may return corrupted ExtraData.extra for non-contiguous input #720
Description
I found an issue in unpackb() while fuzzing Python C extension modules with Atheris and ASan/UBSan enabled.
The sanitizer initially reported memcpy-param-overlap, but the problem is not limited to sanitizer builds. The same input also produces incorrect output with the PyPI wheel, without crashing or emitting a warning.
Environment
- msgpack 1.2.1
- CPython 3.12
- Linux x86_64
Reproducer
import msgpack
# A strided (non-contiguous) view whose logical content is b"\x00abc":
# \x00 -> the integer 0
# abc -> trailing bytes, so unpackb raises ExtraData
buf = bytearray(b"\x00\xffa\xffb\xffc\xff")
view = memoryview(buf)[::2]
assert bytes(view) == b"\x00abc"
try:
msgpack.unpackb(view)
except msgpack.ExtraData as e:
print(e.unpacked) # 0 -- correct
print(e.extra) # b'ab\x00' -- expected b'abc'ASan report
AddressSanitizer: memcpy-param-overlap: memory ranges
[0x7a2549f75340,0x7a2549f75343) and [0x7a2549f75341,0x7a2549f75344) overlap
#1 PyBytes_FromStringAndSize
#2 __pyx_pf_7msgpack_9_cmsgpack_2unpackb msgpack/_unpacker.pyx:199
This looks like a bug to me, but I don't know the codebase well enough to rule out that it's expected for non-contiguous input.
Activity
ThomasWaldmann commentedon Aug 3, 2026
Reproduced on macOS 15 / arm64, CPython 3.14.6, msgpack 1.2.1 (PyPI wheel) — same output as reported, and also on current
main. The pure-Python fallback (MSGPACK_PUREPYTHON=1) is correct; only_cmsgpackis affected. Any non-contiguous view triggers it —[::2]and[::-1]alike.The underlying problem is a use-after-free. For non-contiguous input,
get_data_from_buffer()releases the original view and makes a temporary contiguous copy, sobufpoints into memory owned byview.unpackb()then releasesviewin itsfinallyblock — freeing that copy — and only afterwards readsbuf+offto build theExtraDatapayload.That it is really freed memory, and not just an off-by-one, is easy to show with the macOS allocator's free-poisoning:
So arbitrary heap contents can end up in
ExtraData.extra. Thememcpy-param-overlapASan reports is the same event seen from the other side:PyBytes_FromStringAndSizere-allocates the just-freed block and copies it onto itself.Unpacker.feed()uses the same helper but copies viaappend_buffer()before releasing, so it is unaffected.One more data point: msgpack 1.1.1 does not silently corrupt on this input — it raises
SystemError: <class 'msgpack.exceptions.ExtraData'> returned a result with an exception set(from aBufferError), presumably a different symptom of the pre-#677 buffer handling. So the silent corruption looks specific to 1.2.x; the #677 fix did not cover this path.Fix plus a regression test: #722
ThomasWaldmann commentedon Aug 3, 2026
^ made with Claude Opus 5, carefully review please.
fix: use-after-free in unpackb() ExtraData path for non-contiguous in…