Repository navigation
Conversation
read_var_str() took the length from the stream and used it for mem_alloc(len + 1) and tmp_buf[len] without validation. A negative length wraps to a huge allocation request or indexes before the buffer, and a large length is allocated in full before the short read is detected. Reject negative lengths with E_TSFILE_CORRUPTED and lengths beyond the remaining bytes with E_PARTIAL_READ before allocating. E_PARTIAL_READ is what the truncated read returned before, so chunk/page header parsing keeps retrying with a larger file window. The same guards apply to read_str(std::string&), read_str(String&, pa) and read_mystring(). Valid strings read as before.
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues were identified for the string-deserialization hardening changes.
No open issues 🎉
What changed in this PR
This PR hardens C++ string deserialization by validating decoded lengths before allocation or buffer indexing, while preserving retry behavior for incomplete input.
Changes:
- Reject negative string lengths with
E_TSFILE_CORRUPTED. - Return
E_PARTIAL_READwhen a declared length exceeds the remaining bytes. - Add regression tests for negative and oversized variable-length and fixed-width string lengths.
| File | Description |
|---|---|
cpp/test/common/allocator/byte_stream_test.cc |
Adds malformed-length regression tests and checks that negative lengths leave output unchanged. |
cpp/src/common/allocator/byte_stream.h |
Adds pre-allocation length validation to standard-string and arena-backed string readers. |
Review effort: Balanced
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
String readers used the length from the stream for
mem_allocand buffer indexing without validation. Reject negative lengths withE_TSFILE_CORRUPTEDand lengths beyond the remaining bytes withE_PARTIAL_READ(the code the truncated read returned before, so chunk/page header parsing still retries with a larger file window) before allocating. The same guards apply toread_str(std::string&),read_str(String&, pa)andread_mystring(). Valid strings read as before.Tests:
ByteStreamTest.VarStringRejectsNegativeLength,VarStringRejectsLengthBeyondStream,FixedStringRejectsNegativeLength. Full C++ suite passes.