Repository navigation
feat(types): support GEOMETRY and GEOGRAPHY in Parquet tables - #580
Open
SteNicholas wants to merge 1 commit into
Open
SteNicholas wants to merge 1 commit into
SteNicholas wants to merge 1 commit into
Conversation
SteNicholas
force-pushed
the
PAIMON-578
branch
3 times, most recently
from
November 19, 2026 06:52
24579f3 to
86daea7
Compare
Paimon Java stores geospatial columns as WKB with a logical type, CRS and edge interpolation metadata. C++ could not open these tables; treating their values as BYTES would lose logical attributes and allow pruning by WKB byte ordering. - Add GEOMETRY and GEOGRAPHY schema round trips and public type identification, addressing apache#578. Match Java's defaults, quoted CRS parameters, case-insensitive CRS comparison and five edge algorithms. - Preserve GeoArrow metadata, field IDs and nullability through nested projections, C Data Interface transfers and schema evolution. Reject incompatible casts and geospatial primary, partition, bucket, sequence and sort keys. - Backport native Parquet geospatial annotations to Arrow 17 under C++17, storing WKB in BYTE_ARRAY columns. Require Parquet for data and changelog files, including per-level and operation-specific overrides. - Retain null counts while suppressing WKB min/max statistics and pruning at file, row-group and page level. Treat incomplete counts as unknown and preserve existing BYTES and partition predicate behavior.
SteNicholas
force-pushed
the
PAIMON-578
branch
from
November 19, 2026 07:02
86daea7 to
d9fa2de
Compare
lxy-9602
reviewed
Nov 19, 2026
| # or more contributor license agreements. See the NOTICE file | ||
| # distributed with this work for additional information | ||
| # regarding copyright ownership. The ASF licenses this file | ||
| # to you under the Apache License, Version 2.0 (the |
Member
There was a problem hiding this comment.
Before the review, I think a patch of this large scale (feature-oriented) may not be appropriate. I would recommend rather first upgrade the Arrow version to >21. While, before that, we need performance testing to ensure the new Arrow version doesn’t introduce any performance regression. @lucasfang What do you think?
Member
There was a problem hiding this comment.
I think it would be highly beneficial for the community edition of C++ Paimon to upgrade to C++20. This seems inevitable in the near future. Without it, we won't be able to take advantage of many new features in Parquet, and quite a few excellent third-party libraries now require C++20 or above.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Linked issue: close #578
Support GEOMETRY and GEOGRAPHY columns in Paimon C++, following Paimon Java. Values remain OGC Well-Known Binary (WKB) in Arrow
Binaryarrays, with the logical type, coordinate reference system (CRS) and geography edge interpolation algorithm preserved through schema and data round trips.FieldType::GEOMETRYandFieldType::GEOGRAPHY, and parse and serialize their schema declarations, including quoted CRS parameters, nullability and nested ROW, ARRAY and MAP fields. Match Java's defaultOGC:CRS84CRS, case-insensitive CRS comparison and five geography edge algorithms.geoarrow.wkbfield metadata, field IDs and nullability through the Arrow C Data Interface, nested projections, field mapping, row tracking, data evolution and MAP shredding. Preserve declared MAP key nullability in table schemas while keeping stored keys non-null.Support is initially limited to Parquet. Spatial functions, coordinate transformations and spatial indexes are outside this change.
Tests
API and Format
FieldType::GEOMETRYandFieldType::GEOGRAPHYininclude/paimon/defs.h.Schema::GetFieldTypeidentifies these types from Arrow field metadata.geoarrow.wkbmetadata, and Parquet BYTE_ARRAY columns carry native GEOMETRY/GEOGRAPHY logical annotations compatible with Paimon Java. Existing BYTES columns retain their binary semantics.Documentation
Yes.
docs/source/user_guide/data_types.rstdocuments the geospatial schema syntax, CRS and edge algorithms, Arrow metadata, Parquet storage, nested fields and MAP key nullability, key/order and cast restrictions, statistics and predicates, and the Java/C++ interoperability test.Generative AI tooling
Generated-by: OpenAI Codex (GPT-6)
🤖 Generated with OpenAI Codex