Skip to content

[Feature] Support Vortex file format #319

Closed
1 of 2 issues completed
Closed

Description

@zhouyc-ali

Search before asking

  • I searched in the issues and found nothing similar.

Motivation

Apache Paimon Java supports Vortex through the paimon-vortex module, a Rust-based columnar file format with extensible compression encodings and lazily loaded summary statistics. Paimon C++ cannot currently access tables using this format.

Solution

Add Vortex support to the Paimon C++ file-format SPI:

  • Register the vortex file-format factory.
  • Implement compatible schema, reader, and writer handling.
  • Support column projection, predicate pushdown, and precise row selection.
  • Validate the same supported and unsupported logical types as Java.
  • Add C++ round-trip tests and Java-produced file compatibility tests.

Anything else?

Vortex is implemented in Rust, so the integration can follow the Mosaic Rust FFI patterns (#238).
Java reference module: https://github.com/apache/paimon/tree/master/paimon-vortex

Are you willing to submit a PR?

  • I'm willing to submit a PR!

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions