AVRO-4343: [perl] Fix broken decoder skip path used in schema resolution - #3947
Open
iemejia wants to merge 1 commit into
Open
AVRO-4343: [perl] Fix broken decoder skip path used in schema resolution#3947iemejia wants to merge 1 commit into
iemejia wants to merge 1 commit into
Conversation
The BinaryDecoder skip path (reached when the writer's record has a field
absent from the reader's schema) was broken:
- skip_bytes/skip_string/skip_fixed used seek($n, 0) (SEEK_SET, absolute)
instead of SEEK_CUR, repositioning the reader to an absolute offset and
silently corrupting every subsequent field.
- skip_array/skip_map called skip_block(...) as a plain sub, misaligning
$class/$reader/the callback, so skipping an array or map field crashed
("Can't call method read on unblessed reference"). The negative-block-count
branch also seeked to a negative absolute offset and re-looped without
re-reading the count.
Fix: use relative (SEEK_CUR) skips, reject negative bytes/block sizes, invoke
skip_block as a method, and handle negative block counts like decode_array/
decode_map (abs count + consume block size). Adds regression tests for skipping
non-trailing bytes, arrays, maps, negative-block-count arrays, and rejecting a
negative length.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Fixes the Perl
BinaryDecoderskip path, which is exercised during schema resolution whenever the writer's record contains a field that is absent from the reader's schema (decode_record→skip). Three related defects, present since the initial Perl implementation (AVRO-974):skip_bytes/skip_string/skip_fixeddid$reader->seek($size, 0)— whence0isSEEK_SET(absolute) rather thanSEEK_CUR. Skipping a bytes/string/fixed field repositioned the reader to absolute offset$size, corrupting every subsequent field. (Masked in the existing test only because its skipped field was the last one.)skip_array/skip_mapinvokedskip_block(...)as a plain sub, soshifting$classmisaligned$readerand the content callback, dying with "Can't call method "read" on unblessed reference".skip_blockseeked to a negative absolute offset and re-looped vianextwithout re-reading the block count.Fix
SEEK_CUR) skips inskip_bytes/skip_string/skip_fixed.Avro::Schema::Error::Parse.skip_blockas a method ($class->skip_block), fixing the argument alignment.decode_array/decode_map: take the absolute value and consume the following block-size long.Tests
Adds regression tests to
t/03_bin_decode.tcovering resolution that skips: a non-trailingbytesfield, anarrayfield, amapfield, anarrayencoded with a negative block count, and rejection of a negative bytes length. Full existing Perl suite still passes (the pre-existing04_datafile.tfailure is unrelated — a missingIO::Compressdependency).JIRA: https://issues.apache.org/jira/browse/AVRO-4343