Skip to content

refactor(table): introduce a common table hierarchy - #527

Merged
lxy-9602 merged 7 commits into
apache:mainfrom
lucasfang:codex/table-hierarchy
Nov 13, 2026
Merged

lxy-9602 merged 7 commits into
apache:mainfrom
lucasfang:codex/table-hierarchy

Conversation

@lucasfang

Copy link
Copy Markdown
Member

Purpose

Linked issue: close #526

The current Table::NewScan/NewRead/NewWrite/NewCommit methods route back through context-based factories, so calling an operator on a loaded table does not bind it to that table's schema and storage identity. Table-kind dispatch is also concentrated in table.cpp, with format tables exposed through a separate handle.

Introduce a common table hierarchy so Catalog::GetTable() returns the appropriate managed, format or system table, and operator creation dispatches through that implementation.

  • Add FileStoreTable with append-only and primary-key implementations, reusing the FileStore factories from refactor(table): extract file store factories #516 as merged into main. FileStore continues to derive its branch from options.
  • Make FormatTable and SystemTable implement the common Table interface. System tables expose their own schemas and reject writes and commits.
  • Keep loaded-table operators bound to their schema, path and filesystem while applying operation options and validating conflicting managed-table types, branches and write/commit identities.
  • Route the existing static operator factories through TableFactory and the virtual operator methods. Preserve main's precise file-index evaluation behavior through both entrypoints.
  • Retain catalog ownership for deferred global-system-table reads, and preserve URI routing, fallback-read options, system-table snapshot selection and mandatory changelog settings.

This is the hierarchy step of #459, following #516. Execution-context separation and general Table Copy/time-travel APIs remain separate follow-ups.

Tests

Full build and all 36 CTest targets passed on commit 0d2ddf52:

cmake --build build -j 40
ctest --test-dir build --output-on-failure -j 4
  • Core: 2,682 passed; REST: 190 passed; read integration: 304 passed. Format-specific conditional skips remain in the suite.
  • Focused hierarchy/read/index tests: 223 passed; system-table read integration tests: 32 passed.
  • All applicable pre-commit checks passed for the 56 changed files; git diff apache/main HEAD --check passed.

Regression coverage includes catalog type dispatch, loaded schema/path/filesystem binding, explicit branch conflicts and aliases, builder retries after failed validation, deferred catalog lifetime, URI-map merging for main/fallback reads, changelog sequence numbers, system-table snapshot selection, and precise-index entrypoint parity.

API and Format

  • Table becomes abstract, with a virtual destructor and virtual operator creation methods. Callers that directly constructed Table should load it through the catalog.
  • Rename Table::LatestSchema() and FormatTable::LatestSchema() to GetSchema(), without compatibility aliases. The accessor returns the bound schema without metadata I/O; reload through the catalog to refresh it.
  • Remove CurrentBranch() from the common Table interface; managed-table branch handling belongs to FileStoreTable.
  • FormatTable derives from Table. Catalog::GetFormatTable() remains available for typed callers.
  • The existing ContextBuilder and static operator Create() signatures remain unchanged. Catalogs must be owned by std::shared_ptr when loading global system tables, which retain the catalog for deferred reads.
  • No storage-format or protocol changes.

Documentation

Update the catalog and format-table guides to describe common catalog dispatch, bound-table operator semantics, schema access, branch validation and global-system-table ownership.

Generative AI tooling

Generated-by: OpenAI Codex (GPT-6)

@SteNicholas SteNicholas left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lucasfang, I left two comments for this introduction. PTAL.

Comment thread src/paimon/rest/rest_catalog.cpp Resolved

@zjw1111 zjw1111 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Tests lists build and test commands, pass counts, and pre-commit results for 0d2ddf52 alongside the coverage summary. The PR template uses this section for the UT and IT cases changed by the PR and the behavior they cover.
  • API and Format omits the change to system-table FullName(), which now drops the database name. Uuid() falls back to that value when there is no catalog UUID, so identically named system tables in different databases can return the same fallback ID.
  • Documentation has two stale statements. The system-table guide says sys.tables, sys.partitions, and sys.all_table_options have no public read path. Both catalogs now expose them through a shared catalog handle. The catalog guide says builders reject the identifier spelling tbl$branch_MAIN. The branch resolver now accepts it as an alias for main.

@zjw1111 zjw1111 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few issues on the loaded-table paths, schema reads and code style. See the inline comments.

Comment thread src/paimon/core/table/table_factory.h Resolved
Comment thread include/paimon/catalog/catalog.h Resolved
Preserve query authorization and branch-aware catalog credentials in the table hierarchy. Retain both sets of regression tests and cover authorization through loaded-table virtual operators.

@zjw1111 zjw1111 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The inline comments cover query authorization, format-table identity, fallback reads, and system-table behavior.

Comment thread src/paimon/core/table/table_factory.cpp Resolved

@zjw1111 zjw1111 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. Please address the remaining issues in the follow-up PR.

  • Query authorization still accepts a context from another REST catalog when the table name and URI match. Two storage endpoints can expose the same name and URI, so the scan and reader can still apply B's rules to A's data.
  • An existing $files or $manifests handle can still return raw statistics after REST enables query auth and before the filesystem schema catches up. The read ignores the catalog supplied in the context.
  • A query-auth reader for an already loaded dev table rejects a valid dev split when the read context omits the branch. Split validation uses the context's default main.
  • Two loads of the same REST format table can create different token filesystem wrappers. Comparing their pointers rejects a valid typed write or commit context.
  • Static $ro can still lose options from the on-disk schema when SetTableSchema() is supplied. It can also lose storage settings because the underlying table keeps the filesystem resolved from the context alone.
  • Update the REST format-table example to show GetTable()->NewXxx(), and update the PR's Tests section to describe the added or changed tests and their coverage.

@lxy-9602 lxy-9602 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@lxy-9602
lxy-9602 merged commit 46ffa1d into apache:main Nov 13, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Introduce a common Table hierarchy for managed, format and system tables

4 participants