Skip to content

[python] Parse BlobDescriptor v1/v2 bytes without misclassifying inline payload - #9539

Open
Stephen0421 wants to merge 1 commit into
apache:masterfrom
Stephen0421:pypaimon-blob-pr1a-descriptor
Open

[python] Parse BlobDescriptor v1/v2 bytes without misclassifying inline payload#9539
Stephen0421 wants to merge 1 commit into
apache:masterfrom
Stephen0421:pypaimon-blob-pr1a-descriptor

Conversation

@Stephen0421

Copy link
Copy Markdown
Contributor

Purpose

Align PyPaimon's BlobDescriptor wire format with Java, without changing the existing production read loop.

  • serialize() always writes CURRENT_VERSION (v2 + magic), matching Java BlobDescriptor.serialize().
  • deserialize() accepts v1/v2 prefixes, trailing bytes, and signed URI length (<i, reject negatives).
  • parse_if_serialized is an exact-length parse for v1/v2 (used by writer validation).
  • from_descriptor_bytes is the explicit API for bytes known to be a descriptor. It uses deserialize() (Java prefix contract, trailing padding allowed). It is not a detector: garbage that happens to look like a v1 prefix can yield a BlobRef with a nonsense URI.
  • from_bytes() / is_blob_descriptor stay v2-magic-only so inline blob payload is not classified as a v1 descriptor.
  • blob-descriptor-field writes (write() and write_row()) accept exact v1/v2 bytes and reject trailing padding. Version is checked from data[0] before parse (in [1, CURRENT_VERSION]). Writer validation uses exact wire length instead of a serialize() round-trip, because that round-trip would reject legal v1 after serialize() became always-v2.
  • blob.stored-descriptor-fields is still not a layout switch. Python master ignored that key and wrote dedicated .blob files; a global fallback would mis-parse those tables on a rolling upgrade. The cost is that Java tables which only set the fallback key store inline descriptors, and Python returns those bytes instead of fetching payload. Use blob-descriptor-field (column directives already copy the legacy key onto the canonical option).

Out of scope (follow-up): production convert readers still call Blob.from_bytes(), so historical v1 descriptor columns are stored successfully but not resolved to payload. That path should use from_descriptor_bytes.

Tests

  • pypaimon.tests.blob_test.BlobTest — descriptor serialize/parse, writer v1/v2/trailing/version, from_bytes(allow_blob_data=False) error wrapping
  • pypaimon.tests.blob_table_test.DedicatedFormatWriterTest.test_legacy_stored_descriptor_fields_keeps_dedicated_blob_layout
  • pypaimon.tests.column_directive_utils_test

) from e
# serialize() always emits CURRENT_VERSION, so a round-trip
# would reject exact v1 bytes. Check exact wire length instead.
if BlobDescriptor.parse_if_serialized(descriptor_bytes) is None:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Keep exact VideoFrameDescriptor inputs valid

BlobDescriptor.deserialize() dispatches through BlobDescriptorSerde, so an exact serialized VideoFrameDescriptor is a supported descriptor and the previous round-trip validation accepted it. The new parse_if_serialized() only implements the ordinary v1/v2 BlobDescriptor layouts; for VideoFrameDescriptor("file:///v.mp4", 0, 10, 2).serialize(), deserialization succeeds and reserialization is byte-for-byte equal, but this call returns None, so every such value is now rejected as having trailing bytes. Please dispatch the exact-length check through the serde (or special-case VideoFrameDescriptor with its exact-length deserializer) before applying the ordinary v1/v2 calculation.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thranks. parse_if_serialized() now dispatches through BlobDescriptorSerde: an exact VideoFrameDescriptor is parsed with its exact-length deserializer first, then the ordinary v1/v2 length check. Writer validation still calls BlobDescriptor.parse_if_serialized(), so VideoFrameDescriptor("file:///v.mp4", 0, 10, 2).serialize() is accepted and padded input is still rejected.

…ne payload.

Align serialize() with Java (always CURRENT_VERSION + magic) and add
explicit parse APIs for known descriptor bytes. Writer validation uses
exact wire length so exact v1 input still lands; from_bytes() stays
v2-magic-only so inline blob payload is not treated as a v1 descriptor.
@Stephen0421
Stephen0421 force-pushed the pypaimon-blob-pr1a-descriptor branch from 2cdf837 to 252cb6e Compare September 3, 2026 01:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants