Skip to content

[Feature] Separate table loading and dynamic options from execution contexts #555

Open

Description

@lucasfang

Search before asking

Searched existing issues and reviewed #459 and #533. #459 tracks the broader table refactoring; this issue tracks separating operation Contexts from table loading after #527.

Motivation

The common Table hierarchy is now in main, but Scan/Read/Write/Commit Contexts still mix execution settings with table paths, schemas, filesystems, catalog identity and option maps. A loaded table already binds those resources, so accepting them again through a Context creates competing configuration sources.

The next step should give each input one role: loading parameters select the table, dynamic options configure a table copy, and Contexts describe an operation's execution.

Solution

  • Introduce public TableLoadOptions for static-factory loading: path, explicit filesystem or URI routing, serialized Paimon schema, catalog, identifier and expected UUID.
  • Keep the four Contexts limited to execution settings, such as projection, predicates, caches, executors and commit user.
  • Add Table::Copy(dynamic_options). Copies retain the table's type, location, branch, storage and catalog identity; the original table remains unchanged.
  • Keep Table::NewScan/NewRead/NewWrite/NewCommit context-only. Apply dynamic options through Copy before creating operators.
  • Use Create(load_options, context, dynamic_options) for the static factories, sharing loading and validation.
  • Preserve catalog lifetime, per-table storage credentials, schema caches, query authorization, fallback reads and external format-table directory semantics.
  • Migrate callers, examples, benchmarks, documentation and regression tests.

Anything else?

This follows the separation proposed in #533, rebuilt on main after #527, with dynamic options moved to Table::Copy instead of adding an options argument to NewXXX.

This is a breaking C++ API change. It does not change storage formats or protocols. Branch switching and historical-schema time travel are outside this issue.

Are you willing to submit a PR?

  • I'm willing to submit a PR!
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions