Skip to content

feat(types): support GEOMETRY and GEOGRAPHY in Parquet tables - #580

Open
SteNicholas wants to merge 1 commit into
apache:mainfrom
SteNicholas:PAIMON-578
Open

SteNicholas wants to merge 1 commit into
apache:mainfrom
SteNicholas:PAIMON-578

Conversation

@SteNicholas

Copy link
Copy Markdown
Member

Purpose

Linked issue: close #578

Support GEOMETRY and GEOGRAPHY columns in Paimon C++, following Paimon Java. Values remain OGC Well-Known Binary (WKB) in Arrow Binary arrays, with the logical type, coordinate reference system (CRS) and geography edge interpolation algorithm preserved through schema and data round trips.

  • Add FieldType::GEOMETRY and FieldType::GEOGRAPHY, and parse and serialize their schema declarations, including quoted CRS parameters, nullability and nested ROW, ARRAY and MAP fields. Match Java's default OGC:CRS84 CRS, case-insensitive CRS comparison and five geography edge algorithms.
  • Preserve geoarrow.wkb field metadata, field IDs and nullability through the Arrow C Data Interface, nested projections, field mapping, row tracking, data evolution and MAP shredding. Preserve declared MAP key nullability in table schemas while keeping stored keys non-null.
  • Make type comparison, casts and schema evolution distinguish BYTES, GEOMETRY, GEOGRAPHY, different CRS values and different edge algorithms. Reject geospatial primary, partition and bucket keys, sequence fields and sequence-group ordering keys, and clustering/sort columns.
  • Backport native Parquet GEOMETRY/GEOGRAPHY annotations to Arrow 17 while retaining C++17. Store WKB in BYTE_ARRAY columns and preserve the annotations and parameters for nested columns, including Java-compatible LIST element and MAP key/value field IDs.
  • Require Parquet data and changelog files, including per-level formats and operation-specific overrides, and reject geospatial schemas at unsupported format readers and writers.
  • Retain null counts while suppressing WKB min/max statistics in Paimon, the realtime store and Parquet. Ignore WKB bounds in file, row-group and page pruning, treat incomplete null counts as unknown, and support null predicates without changing ordinary BYTES or partition predicate behavior.

Support is initially limited to Parquet. Spatial functions, coordinate transformations and spatial indexes are outside this change.

Tests

  • Schema and type tests for default and custom CRS values, quoted and Unicode CRS parameters, all five geography edge algorithms, invalid metadata, logical equality, incompatible casts and schema evolution, and nested C Data Interface round trips.
  • Metadata preservation tests for projections, field mapping, MAP shredding, data evolution, row tracking and BLOB view resolution; validation tests for unsupported key/order usages and file formats, including per-level and operation-specific overrides.
  • Statistics and predicate tests for retained null counts, absent geospatial min/max bounds, incomplete row-group statistics, null predicates, rejected comparisons, page pruning and unchanged BYTES behavior.
  • Parquet tests for native annotations and field IDs, nested ROW/ARRAY/MAP values, nulls and valid empty geometries, sliced array adaptation, incompatible read schemas, default native parameters and escaped CRS round trips.
  • Integration tests for table reads with reordered projections and null predicates, and Java-to-C++ and C++-to-Java Parquet interoperability covering WKB values, logical parameters, nested annotations and statistics.

API and Format

  • API: adds FieldType::GEOMETRY and FieldType::GEOGRAPHY in include/paimon/defs.h. Schema::GetFieldType identifies these types from Arrow field metadata.
  • Format: table schemas preserve GEOMETRY/GEOGRAPHY declarations and parameters. Arrow fields carry geoarrow.wkb metadata, and Parquet BYTE_ARRAY columns carry native GEOMETRY/GEOGRAPHY logical annotations compatible with Paimon Java. Existing BYTES columns retain their binary semantics.

Documentation

Yes. docs/source/user_guide/data_types.rst documents the geospatial schema syntax, CRS and edge algorithms, Arrow metadata, Parquet storage, nested fields and MAP key nullability, key/order and cast restrictions, statistics and predicates, and the Java/C++ interoperability test.

Generative AI tooling

Generated-by: OpenAI Codex (GPT-6)

🤖 Generated with OpenAI Codex

@SteNicholas
SteNicholas force-pushed the PAIMON-578 branch 3 times, most recently from 24579f3 to 86daea7 Compare November 19, 2026 06:52
Paimon Java stores geospatial columns as WKB with a logical type, CRS
and edge interpolation metadata. C++ could not open these tables;
treating their values as BYTES would lose logical attributes and allow
pruning by WKB byte ordering.

- Add GEOMETRY and GEOGRAPHY schema round trips and public type
  identification, addressing apache#578. Match Java's defaults, quoted CRS
  parameters, case-insensitive CRS comparison and five edge algorithms.
- Preserve GeoArrow metadata, field IDs and nullability through nested
  projections, C Data Interface transfers and schema evolution. Reject
  incompatible casts and geospatial primary, partition, bucket, sequence
  and sort keys.
- Backport native Parquet geospatial annotations to Arrow 17 under
  C++17, storing WKB in BYTE_ARRAY columns. Require Parquet for data and
  changelog files, including per-level and operation-specific overrides.
- Retain null counts while suppressing WKB min/max statistics and
  pruning at file, row-group and page level. Treat incomplete counts as
  unknown and preserve existing BYTES and partition predicate behavior.
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before the review, I think a patch of this large scale (feature-oriented) may not be appropriate. I would recommend rather first upgrade the Arrow version to >21. While, before that, we need performance testing to ensure the new Arrow version doesn’t introduce any performance regression. @lucasfang What do you think?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be highly beneficial for the community edition of C++ Paimon to upgrade to C++20. This seems inevitable in the near future. Without it, we won't be able to take advantage of many new features in Parquet, and quite a few excellent third-party libraries now require C++20 or above.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Support GEOMETRY and GEOGRAPHY types

3 participants