[python] Parse BlobDescriptor v1/v2 bytes without misclassifying inline payload - #9539
[python] Parse BlobDescriptor v1/v2 bytes without misclassifying inline payload#9539Stephen0421 wants to merge 1 commit into
Conversation
| ) from e | ||
| # serialize() always emits CURRENT_VERSION, so a round-trip | ||
| # would reject exact v1 bytes. Check exact wire length instead. | ||
| if BlobDescriptor.parse_if_serialized(descriptor_bytes) is None: |
There was a problem hiding this comment.
[P2] Keep exact VideoFrameDescriptor inputs valid
BlobDescriptor.deserialize() dispatches through BlobDescriptorSerde, so an exact serialized VideoFrameDescriptor is a supported descriptor and the previous round-trip validation accepted it. The new parse_if_serialized() only implements the ordinary v1/v2 BlobDescriptor layouts; for VideoFrameDescriptor("file:///v.mp4", 0, 10, 2).serialize(), deserialization succeeds and reserialization is byte-for-byte equal, but this call returns None, so every such value is now rejected as having trailing bytes. Please dispatch the exact-length check through the serde (or special-case VideoFrameDescriptor with its exact-length deserializer) before applying the ordinary v1/v2 calculation.
There was a problem hiding this comment.
Thranks. parse_if_serialized() now dispatches through BlobDescriptorSerde: an exact VideoFrameDescriptor is parsed with its exact-length deserializer first, then the ordinary v1/v2 length check. Writer validation still calls BlobDescriptor.parse_if_serialized(), so VideoFrameDescriptor("file:///v.mp4", 0, 10, 2).serialize() is accepted and padded input is still rejected.
…ne payload. Align serialize() with Java (always CURRENT_VERSION + magic) and add explicit parse APIs for known descriptor bytes. Writer validation uses exact wire length so exact v1 input still lands; from_bytes() stays v2-magic-only so inline blob payload is not treated as a v1 descriptor.
2cdf837 to
252cb6e
Compare
Purpose
Align PyPaimon's BlobDescriptor wire format with Java, without changing the existing production read loop.
serialize()always writesCURRENT_VERSION(v2 + magic), matching JavaBlobDescriptor.serialize().deserialize()accepts v1/v2 prefixes, trailing bytes, and signed URI length (<i, reject negatives).parse_if_serializedis an exact-length parse for v1/v2 (used by writer validation).from_descriptor_bytesis the explicit API for bytes known to be a descriptor. It usesdeserialize()(Java prefix contract, trailing padding allowed). It is not a detector: garbage that happens to look like a v1 prefix can yield a BlobRef with a nonsense URI.from_bytes()/is_blob_descriptorstay v2-magic-only so inline blob payload is not classified as a v1 descriptor.blob-descriptor-fieldwrites (write()andwrite_row()) accept exact v1/v2 bytes and reject trailing padding. Version is checked fromdata[0]before parse (in [1, CURRENT_VERSION]). Writer validation uses exact wire length instead of aserialize()round-trip, because that round-trip would reject legal v1 afterserialize()became always-v2.blob.stored-descriptor-fieldsis still not a layout switch. Python master ignored that key and wrote dedicated.blobfiles; a global fallback would mis-parse those tables on a rolling upgrade. The cost is that Java tables which only set the fallback key store inline descriptors, and Python returns those bytes instead of fetching payload. Useblob-descriptor-field(column directives already copy the legacy key onto the canonical option).Out of scope (follow-up): production convert readers still call
Blob.from_bytes(), so historical v1 descriptor columns are stored successfully but not resolved to payload. That path should usefrom_descriptor_bytes.Tests
pypaimon.tests.blob_test.BlobTest— descriptor serialize/parse, writer v1/v2/trailing/version,from_bytes(allow_blob_data=False)error wrappingpypaimon.tests.blob_table_test.DedicatedFormatWriterTest.test_legacy_stored_descriptor_fields_keeps_dedicated_blob_layoutpypaimon.tests.column_directive_utils_test