Repository navigation
Conversation
_deserialize_bitmap read an 8-byte count and used it directly as a loop bound, so a small blob could declare a large number of bitmaps. Reading past the payload surfaced as an IndexError from the native bitmap decoder. Validate the count against the remaining bytes, since every bitmap contributes at least a 4-byte key.
|
Checking in on this one, and offering to close out the question in the description. The PR rejects a bitmap count larger than the payload. It deliberately leaves one adjacent case alone: a single large key still drives the empty-bitmap fill loop up to Happy to go either way — fold that into this PR, or send it separately once this lands. If you would rather it were one change, say so and I will extend this one instead of leaving the question open. |
|
Still current: CI is green and the branch merges cleanly against main. The open question from the description (whether the large-key fill loop should be bounded here or in a follow-up) still stands, and I am happy to go either way. |
Closes #3979
Rationale for this change
_deserialize_bitmapread an 8-byte count from the start of the payload and used it directly as a loop bound, without relating it to how much payload actually follows. A small blob could therefore declare a very large number of bitmaps.The count cannot be trusted on its own, but it can be checked against the payload: every bitmap contributes at least a 4-byte key, so a count larger than
len(payload) // 4cannot be satisfied by the bytes present. This rejects such a payload up front with aValueErrorinstead of letting the loop run off the end of the buffer, which surfaced asIndexError: Out of bounds on buffer access (axis 0)from the native bitmap decoder.Per
SECURITY-THREAT-MODEL.mdthis is parser hardening (§7) rather than a vulnerability, so it is filed as an ordinary bug fix.Are these changes tested?
Yes.
tests/table/test_deletion_vector.pygains a case for a payload that declares more bitmaps than its bytes can hold. It fails onmainwith theIndexErrorabove and passes with this change; the existing deletion-vector tests are unaffected.Are there any user-facing changes?
A deletion vector whose declared bitmap count exceeds its payload now raises
ValueErrorinstead ofIndexError. Valid deletion vectors are unaffected.One related case is deliberately left out of this PR to keep it to a single concern: a single large key still drives the empty-bitmap fill loop up to
MAX_JAVA_SIGNEDappends, which is the same shape of problem reached through a different field. Happy to follow up on that separately if you would like it addressed.